2023-04-22 14:53:32,033 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure timeout: 13 mins 2023-04-22 14:53:32,329 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3 2023-04-22 14:53:32,346 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-22 14:53:32,346 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a, deleteOnExit=true 2023-04-22 14:53:32,346 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-22 14:53:32,347 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/test.cache.data in system properties and HBase conf 2023-04-22 14:53:32,347 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/hadoop.tmp.dir in system properties and HBase conf 2023-04-22 14:53:32,347 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/hadoop.log.dir in system properties and HBase conf 2023-04-22 14:53:32,348 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-22 14:53:32,349 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-22 14:53:32,349 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-22 14:53:32,473 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-22 14:53:32,904 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-22 14:53:32,910 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-22 14:53:32,910 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-22 14:53:32,911 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-22 14:53:32,911 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-22 14:53:32,912 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-22 14:53:32,912 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-22 14:53:32,913 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-22 14:53:32,913 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-22 14:53:32,914 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-22 14:53:32,914 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/nfs.dump.dir in system properties and HBase conf 2023-04-22 14:53:32,914 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/java.io.tmpdir in system properties and HBase conf 2023-04-22 14:53:32,915 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-22 14:53:32,915 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-22 14:53:32,916 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-22 14:53:33,509 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-22 14:53:33,514 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-22 14:53:33,817 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-22 14:53:34,013 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-22 14:53:34,027 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-22 14:53:34,066 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-22 14:53:34,103 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/java.io.tmpdir/Jetty_localhost_45969_hdfs____.3jpupn/webapp 2023-04-22 14:53:34,289 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45969 2023-04-22 14:53:34,300 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-22 14:53:34,300 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-22 14:53:34,855 WARN [Listener at localhost/35843] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-22 14:53:34,931 WARN [Listener at localhost/35843] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-22 14:53:34,953 WARN [Listener at localhost/35843] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-22 14:53:34,960 INFO [Listener at localhost/35843] log.Slf4jLog(67): jetty-6.1.26 2023-04-22 14:53:34,966 INFO [Listener at localhost/35843] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/java.io.tmpdir/Jetty_localhost_38099_datanode____hdu5o2/webapp 2023-04-22 14:53:35,065 INFO [Listener at localhost/35843] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38099 2023-04-22 14:53:35,442 WARN [Listener at localhost/44223] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-22 14:53:35,461 WARN [Listener at localhost/44223] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-22 14:53:35,468 WARN [Listener at localhost/44223] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-22 14:53:35,472 INFO [Listener at localhost/44223] log.Slf4jLog(67): jetty-6.1.26 2023-04-22 14:53:35,481 INFO [Listener at localhost/44223] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/java.io.tmpdir/Jetty_localhost_41945_datanode____tbg7d0/webapp 2023-04-22 14:53:35,607 INFO [Listener at localhost/44223] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41945 2023-04-22 14:53:35,626 WARN [Listener at localhost/37111] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-22 14:53:35,678 WARN [Listener at localhost/37111] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-22 14:53:35,682 WARN [Listener at localhost/37111] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-22 14:53:35,683 INFO [Listener at localhost/37111] log.Slf4jLog(67): jetty-6.1.26 2023-04-22 14:53:35,697 INFO [Listener at localhost/37111] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/java.io.tmpdir/Jetty_localhost_43497_datanode____.lodq92/webapp 2023-04-22 14:53:35,848 INFO [Listener at localhost/37111] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43497 2023-04-22 14:53:35,884 WARN [Listener at localhost/43403] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-22 14:53:36,089 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf583803cbf8e7570: Processing first storage report for DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6 from datanode 6cc12795-ed03-4ce7-985d-18d7c9a6ec2b 2023-04-22 14:53:36,091 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf583803cbf8e7570: from storage DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6 node DatanodeRegistration(127.0.0.1:41961, datanodeUuid=6cc12795-ed03-4ce7-985d-18d7c9a6ec2b, infoPort=46381, infoSecurePort=0, ipcPort=37111, storageInfo=lv=-57;cid=testClusterID;nsid=1696505568;c=1682175213588), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-04-22 14:53:36,091 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc3d4349519b27b55: Processing first storage report for DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c from datanode cd7c3aae-8b07-4809-9172-b4e8dd2fdabd 2023-04-22 14:53:36,091 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc3d4349519b27b55: from storage DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c node DatanodeRegistration(127.0.0.1:35001, datanodeUuid=cd7c3aae-8b07-4809-9172-b4e8dd2fdabd, infoPort=33585, infoSecurePort=0, ipcPort=44223, storageInfo=lv=-57;cid=testClusterID;nsid=1696505568;c=1682175213588), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-22 14:53:36,091 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6ddac1b5cb7490b8: Processing first storage report for DS-8329503a-6d5e-45e3-9091-b3a05c845e79 from datanode 0e9964a3-eb0e-4f0a-bd76-2c6599875fd5 2023-04-22 14:53:36,092 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6ddac1b5cb7490b8: from storage DS-8329503a-6d5e-45e3-9091-b3a05c845e79 node DatanodeRegistration(127.0.0.1:42675, datanodeUuid=0e9964a3-eb0e-4f0a-bd76-2c6599875fd5, infoPort=44303, infoSecurePort=0, ipcPort=43403, storageInfo=lv=-57;cid=testClusterID;nsid=1696505568;c=1682175213588), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-22 14:53:36,092 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc3d4349519b27b55: Processing first storage report for DS-ad03e83b-71e8-40a8-be50-4fec99059a97 from datanode cd7c3aae-8b07-4809-9172-b4e8dd2fdabd 2023-04-22 14:53:36,092 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc3d4349519b27b55: from storage DS-ad03e83b-71e8-40a8-be50-4fec99059a97 node DatanodeRegistration(127.0.0.1:35001, datanodeUuid=cd7c3aae-8b07-4809-9172-b4e8dd2fdabd, infoPort=33585, infoSecurePort=0, ipcPort=44223, storageInfo=lv=-57;cid=testClusterID;nsid=1696505568;c=1682175213588), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-22 14:53:36,092 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x6ddac1b5cb7490b8: Processing first storage report for DS-dfd91d62-858c-414a-9a75-39def1070355 from datanode 0e9964a3-eb0e-4f0a-bd76-2c6599875fd5 2023-04-22 14:53:36,092 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x6ddac1b5cb7490b8: from storage DS-dfd91d62-858c-414a-9a75-39def1070355 node DatanodeRegistration(127.0.0.1:42675, datanodeUuid=0e9964a3-eb0e-4f0a-bd76-2c6599875fd5, infoPort=44303, infoSecurePort=0, ipcPort=43403, storageInfo=lv=-57;cid=testClusterID;nsid=1696505568;c=1682175213588), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-22 14:53:36,093 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xf583803cbf8e7570: Processing first storage report for DS-6f6e9b56-ab11-4214-b86c-a8bd3b4845a1 from datanode 6cc12795-ed03-4ce7-985d-18d7c9a6ec2b 2023-04-22 14:53:36,093 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xf583803cbf8e7570: from storage DS-6f6e9b56-ab11-4214-b86c-a8bd3b4845a1 node DatanodeRegistration(127.0.0.1:41961, datanodeUuid=6cc12795-ed03-4ce7-985d-18d7c9a6ec2b, infoPort=46381, infoSecurePort=0, ipcPort=37111, storageInfo=lv=-57;cid=testClusterID;nsid=1696505568;c=1682175213588), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-22 14:53:36,363 DEBUG [Listener at localhost/43403] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3 2023-04-22 14:53:36,433 INFO [Listener at localhost/43403] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/zookeeper_0, clientPort=50713, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-22 14:53:36,448 INFO [Listener at localhost/43403] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=50713 2023-04-22 14:53:36,460 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:36,463 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:37,129 INFO [Listener at localhost/43403] util.FSUtils(471): Created version file at hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd with version=8 2023-04-22 14:53:37,129 INFO [Listener at localhost/43403] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/hbase-staging 2023-04-22 14:53:37,446 INFO [Listener at localhost/43403] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-22 14:53:37,954 INFO [Listener at localhost/43403] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-22 14:53:37,986 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:37,987 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:37,987 INFO [Listener at localhost/43403] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-22 14:53:37,988 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:37,988 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-22 14:53:38,140 INFO [Listener at localhost/43403] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-22 14:53:38,220 DEBUG [Listener at localhost/43403] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-22 14:53:38,323 INFO [Listener at localhost/43403] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43497 2023-04-22 14:53:38,333 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,335 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,358 INFO [Listener at localhost/43403] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43497 connecting to ZooKeeper ensemble=127.0.0.1:50713 2023-04-22 14:53:38,405 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:434970x0, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-22 14:53:38,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43497-0x101ac5cb4e00000 connected 2023-04-22 14:53:38,437 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 14:53:38,438 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:53:38,441 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-22 14:53:38,450 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43497 2023-04-22 14:53:38,450 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43497 2023-04-22 14:53:38,451 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43497 2023-04-22 14:53:38,452 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43497 2023-04-22 14:53:38,452 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43497 2023-04-22 14:53:38,458 INFO [Listener at localhost/43403] master.HMaster(444): hbase.rootdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd, hbase.cluster.distributed=false 2023-04-22 14:53:38,530 INFO [Listener at localhost/43403] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-22 14:53:38,530 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,530 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,530 INFO [Listener at localhost/43403] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-22 14:53:38,530 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,530 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-22 14:53:38,535 INFO [Listener at localhost/43403] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-22 14:53:38,539 INFO [Listener at localhost/43403] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35417 2023-04-22 14:53:38,541 INFO [Listener at localhost/43403] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-22 14:53:38,549 DEBUG [Listener at localhost/43403] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-22 14:53:38,550 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,552 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,555 INFO [Listener at localhost/43403] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35417 connecting to ZooKeeper ensemble=127.0.0.1:50713 2023-04-22 14:53:38,561 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:354170x0, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-22 14:53:38,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35417-0x101ac5cb4e00001 connected 2023-04-22 14:53:38,562 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 14:53:38,564 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:53:38,565 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-22 14:53:38,565 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35417 2023-04-22 14:53:38,566 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35417 2023-04-22 14:53:38,566 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35417 2023-04-22 14:53:38,567 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35417 2023-04-22 14:53:38,568 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35417 2023-04-22 14:53:38,584 INFO [Listener at localhost/43403] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-22 14:53:38,584 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,585 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,585 INFO [Listener at localhost/43403] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-22 14:53:38,586 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,586 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-22 14:53:38,586 INFO [Listener at localhost/43403] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-22 14:53:38,589 INFO [Listener at localhost/43403] ipc.NettyRpcServer(120): Bind to /172.31.14.131:36267 2023-04-22 14:53:38,590 INFO [Listener at localhost/43403] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-22 14:53:38,591 DEBUG [Listener at localhost/43403] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-22 14:53:38,592 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,594 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,597 INFO [Listener at localhost/43403] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:36267 connecting to ZooKeeper ensemble=127.0.0.1:50713 2023-04-22 14:53:38,603 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:362670x0, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-22 14:53:38,605 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): regionserver:362670x0, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 14:53:38,606 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): regionserver:362670x0, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:53:38,607 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): regionserver:362670x0, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-22 14:53:38,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:36267-0x101ac5cb4e00002 connected 2023-04-22 14:53:38,608 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=36267 2023-04-22 14:53:38,611 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=36267 2023-04-22 14:53:38,615 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=36267 2023-04-22 14:53:38,616 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=36267 2023-04-22 14:53:38,618 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=36267 2023-04-22 14:53:38,632 INFO [Listener at localhost/43403] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-22 14:53:38,632 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,632 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,633 INFO [Listener at localhost/43403] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-22 14:53:38,633 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-22 14:53:38,633 INFO [Listener at localhost/43403] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-22 14:53:38,633 INFO [Listener at localhost/43403] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-22 14:53:38,636 INFO [Listener at localhost/43403] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33597 2023-04-22 14:53:38,637 INFO [Listener at localhost/43403] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-22 14:53:38,638 DEBUG [Listener at localhost/43403] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-22 14:53:38,639 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,641 INFO [Listener at localhost/43403] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,642 INFO [Listener at localhost/43403] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:33597 connecting to ZooKeeper ensemble=127.0.0.1:50713 2023-04-22 14:53:38,647 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:335970x0, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-22 14:53:38,648 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): regionserver:335970x0, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 14:53:38,649 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): regionserver:335970x0, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:53:38,650 DEBUG [Listener at localhost/43403] zookeeper.ZKUtil(164): regionserver:335970x0, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-22 14:53:38,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:33597-0x101ac5cb4e00003 connected 2023-04-22 14:53:38,653 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33597 2023-04-22 14:53:38,653 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33597 2023-04-22 14:53:38,653 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33597 2023-04-22 14:53:38,656 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33597 2023-04-22 14:53:38,656 DEBUG [Listener at localhost/43403] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33597 2023-04-22 14:53:38,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:53:38,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-22 14:53:38,671 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:53:38,693 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-22 14:53:38,693 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-22 14:53:38,693 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-22 14:53:38,693 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-22 14:53:38,694 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:38,695 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-22 14:53:38,697 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,43497,1682175217266 from backup master directory 2023-04-22 14:53:38,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-22 14:53:38,702 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:53:38,702 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-22 14:53:38,702 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-22 14:53:38,703 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:53:38,706 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-22 14:53:38,707 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-22 14:53:38,832 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/hbase.id with ID: febd5dd1-fb6f-4e72-a2a0-a2b783c895a7 2023-04-22 14:53:38,890 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-22 14:53:38,911 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:38,964 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x27b5d011 to 127.0.0.1:50713 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:39,002 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ab9852, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:39,028 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:53:39,030 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-22 14:53:39,049 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-22 14:53:39,049 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-22 14:53:39,051 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:53:39,057 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:53:39,058 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 14:53:39,106 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/data/master/store-tmp 2023-04-22 14:53:39,154 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:39,154 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-22 14:53:39,154 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:53:39,155 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:53:39,155 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-22 14:53:39,155 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:53:39,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:53:39,155 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-22 14:53:39,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/WALs/jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:53:39,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43497%2C1682175217266, suffix=, logDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/WALs/jenkins-hbase4.apache.org,43497,1682175217266, archiveDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/oldWALs, maxLogs=10 2023-04-22 14:53:39,255 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK] 2023-04-22 14:53:39,255 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK] 2023-04-22 14:53:39,255 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK] 2023-04-22 14:53:39,266 DEBUG [RS-EventLoopGroup-5-3] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:53:39,354 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/WALs/jenkins-hbase4.apache.org,43497,1682175217266/jenkins-hbase4.apache.org%2C43497%2C1682175217266.1682175219188 2023-04-22 14:53:39,360 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK], DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK], DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK]] 2023-04-22 14:53:39,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:39,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:39,365 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:39,367 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:39,436 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:39,443 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-22 14:53:39,469 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-22 14:53:39,482 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:39,488 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:39,490 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:39,510 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-22 14:53:39,515 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:39,516 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70933158, jitterRate=0.05698642134666443}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-22 14:53:39,516 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-22 14:53:39,518 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-22 14:53:39,543 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:39,543 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:39,546 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:39,548 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-22 14:53:39,581 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-04-22 14:53:39,581 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-22 14:53:39,606 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-22 14:53:39,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-22 14:53:39,635 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-22 14:53:39,639 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-22 14:53:39,641 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-22 14:53:39,648 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-22 14:53:39,653 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-22 14:53:39,656 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:39,658 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-22 14:53:39,659 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-22 14:53:39,672 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-22 14:53:39,678 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-22 14:53:39,678 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-22 14:53:39,678 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-22 14:53:39,678 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:39,678 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-22 14:53:39,679 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,43497,1682175217266, sessionid=0x101ac5cb4e00000, setting cluster-up flag (Was=false) 2023-04-22 14:53:39,699 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:39,705 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-22 14:53:39,707 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:53:39,711 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:39,721 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-22 14:53:39,723 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:53:39,725 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.hbase-snapshot/.tmp 2023-04-22 14:53:39,761 INFO [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(952): ClusterId : febd5dd1-fb6f-4e72-a2a0-a2b783c895a7 2023-04-22 14:53:39,761 INFO [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(952): ClusterId : febd5dd1-fb6f-4e72-a2a0-a2b783c895a7 2023-04-22 14:53:39,761 INFO [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(952): ClusterId : febd5dd1-fb6f-4e72-a2a0-a2b783c895a7 2023-04-22 14:53:39,766 DEBUG [RS:1;jenkins-hbase4:36267] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-22 14:53:39,766 DEBUG [RS:2;jenkins-hbase4:33597] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-22 14:53:39,766 DEBUG [RS:0;jenkins-hbase4:35417] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-22 14:53:39,773 DEBUG [RS:2;jenkins-hbase4:33597] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-22 14:53:39,773 DEBUG [RS:0;jenkins-hbase4:35417] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-22 14:53:39,773 DEBUG [RS:1;jenkins-hbase4:36267] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-22 14:53:39,774 DEBUG [RS:0;jenkins-hbase4:35417] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-22 14:53:39,773 DEBUG [RS:2;jenkins-hbase4:33597] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-22 14:53:39,774 DEBUG [RS:1;jenkins-hbase4:36267] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-22 14:53:39,779 DEBUG [RS:0;jenkins-hbase4:35417] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-22 14:53:39,779 DEBUG [RS:1;jenkins-hbase4:36267] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-22 14:53:39,781 DEBUG [RS:0;jenkins-hbase4:35417] zookeeper.ReadOnlyZKClient(139): Connect 0x386efff5 to 127.0.0.1:50713 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:39,782 DEBUG [RS:2;jenkins-hbase4:33597] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-22 14:53:39,784 DEBUG [RS:1;jenkins-hbase4:36267] zookeeper.ReadOnlyZKClient(139): Connect 0x2e7fce47 to 127.0.0.1:50713 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:39,784 DEBUG [RS:2;jenkins-hbase4:33597] zookeeper.ReadOnlyZKClient(139): Connect 0x359d0571 to 127.0.0.1:50713 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:39,821 DEBUG [RS:2;jenkins-hbase4:33597] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b5aedb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:39,822 DEBUG [RS:0;jenkins-hbase4:35417] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@36a90688, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:39,822 DEBUG [RS:2;jenkins-hbase4:33597] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f656e01, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-22 14:53:39,822 DEBUG [RS:0;jenkins-hbase4:35417] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32969485, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-22 14:53:39,825 DEBUG [RS:1;jenkins-hbase4:36267] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20b825f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:39,825 DEBUG [RS:1;jenkins-hbase4:36267] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2487a537, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-22 14:53:39,851 DEBUG [RS:2;jenkins-hbase4:33597] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:33597 2023-04-22 14:53:39,853 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35417 2023-04-22 14:53:39,854 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:36267 2023-04-22 14:53:39,857 INFO [RS:0;jenkins-hbase4:35417] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-22 14:53:39,857 INFO [RS:2;jenkins-hbase4:33597] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-22 14:53:39,857 INFO [RS:1;jenkins-hbase4:36267] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-22 14:53:39,858 INFO [RS:1;jenkins-hbase4:36267] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-22 14:53:39,858 INFO [RS:2;jenkins-hbase4:33597] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-22 14:53:39,858 INFO [RS:0;jenkins-hbase4:35417] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-22 14:53:39,858 DEBUG [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(1023): About to register with Master. 2023-04-22 14:53:39,858 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1023): About to register with Master. 2023-04-22 14:53:39,858 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1023): About to register with Master. 2023-04-22 14:53:39,861 INFO [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43497,1682175217266 with isa=jenkins-hbase4.apache.org/172.31.14.131:33597, startcode=1682175218631 2023-04-22 14:53:39,861 INFO [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43497,1682175217266 with isa=jenkins-hbase4.apache.org/172.31.14.131:36267, startcode=1682175218583 2023-04-22 14:53:39,861 INFO [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43497,1682175217266 with isa=jenkins-hbase4.apache.org/172.31.14.131:35417, startcode=1682175218529 2023-04-22 14:53:39,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:39,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 14:53:39,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 14:53:39,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 14:53:39,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-22 14:53:39,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-22 14:53:39,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-22 14:53:39,876 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:39,877 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175249877 2023-04-22 14:53:39,880 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-22 14:53:39,880 DEBUG [RS:1;jenkins-hbase4:36267] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-22 14:53:39,880 DEBUG [RS:0;jenkins-hbase4:35417] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-22 14:53:39,880 DEBUG [RS:2;jenkins-hbase4:33597] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-22 14:53:39,886 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:39,886 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-22 14:53:39,893 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-22 14:53:39,894 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-22 14:53:39,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-22 14:53:39,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-22 14:53:39,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-22 14:53:39,902 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-22 14:53:39,903 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,905 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-22 14:53:39,907 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-22 14:53:39,908 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-22 14:53:39,913 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-22 14:53:39,915 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-22 14:53:39,917 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682175219917,5,FailOnTimeoutGroup] 2023-04-22 14:53:39,918 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682175219917,5,FailOnTimeoutGroup] 2023-04-22 14:53:39,918 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,919 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-22 14:53:39,920 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,920 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:39,945 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:39,947 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:39,947 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd 2023-04-22 14:53:39,952 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40693, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-04-22 14:53:39,952 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49461, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-22 14:53:39,952 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41245, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-04-22 14:53:39,975 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=43497] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:39,976 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43497] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:39,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:39,977 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:39,979 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-22 14:53:39,982 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/info 2023-04-22 14:53:39,983 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-22 14:53:39,984 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:39,984 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-22 14:53:39,987 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/rep_barrier 2023-04-22 14:53:39,990 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-22 14:53:39,991 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:39,991 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-22 14:53:39,993 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/table 2023-04-22 14:53:39,994 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd 2023-04-22 14:53:39,994 DEBUG [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd 2023-04-22 14:53:39,995 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-22 14:53:39,994 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd 2023-04-22 14:53:39,995 DEBUG [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35843 2023-04-22 14:53:39,995 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35843 2023-04-22 14:53:39,996 DEBUG [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-22 14:53:39,995 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:35843 2023-04-22 14:53:39,996 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-22 14:53:39,996 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-22 14:53:39,996 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:39,998 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740 2023-04-22 14:53:39,998 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740 2023-04-22 14:53:40,003 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-04-22 14:53:40,005 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:53:40,006 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-22 14:53:40,007 DEBUG [RS:1;jenkins-hbase4:36267] zookeeper.ZKUtil(162): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:40,007 WARN [RS:1;jenkins-hbase4:36267] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-22 14:53:40,007 DEBUG [RS:2;jenkins-hbase4:33597] zookeeper.ZKUtil(162): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:40,007 INFO [RS:1;jenkins-hbase4:36267] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 14:53:40,007 DEBUG [RS:0;jenkins-hbase4:35417] zookeeper.ZKUtil(162): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:40,008 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:40,007 WARN [RS:2;jenkins-hbase4:33597] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-22 14:53:40,008 WARN [RS:0;jenkins-hbase4:35417] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-22 14:53:40,008 INFO [RS:2;jenkins-hbase4:33597] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 14:53:40,008 INFO [RS:0;jenkins-hbase4:35417] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 14:53:40,009 DEBUG [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:40,009 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1947): logDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:40,014 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,36267,1682175218583] 2023-04-22 14:53:40,014 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35417,1682175218529] 2023-04-22 14:53:40,014 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,33597,1682175218631] 2023-04-22 14:53:40,021 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:40,023 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67456702, jitterRate=0.005183190107345581}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-04-22 14:53:40,023 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-22 14:53:40,024 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-22 14:53:40,024 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-22 14:53:40,024 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-22 14:53:40,025 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 1 ms 2023-04-22 14:53:40,025 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-22 14:53:40,026 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-22 14:53:40,027 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-22 14:53:40,033 DEBUG [RS:2;jenkins-hbase4:33597] zookeeper.ZKUtil(162): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:40,033 DEBUG [RS:1;jenkins-hbase4:36267] zookeeper.ZKUtil(162): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:40,034 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:40,034 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-22 14:53:40,034 DEBUG [RS:2;jenkins-hbase4:33597] zookeeper.ZKUtil(162): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:40,034 DEBUG [RS:0;jenkins-hbase4:35417] zookeeper.ZKUtil(162): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:40,035 DEBUG [RS:1;jenkins-hbase4:36267] zookeeper.ZKUtil(162): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:40,035 DEBUG [RS:0;jenkins-hbase4:35417] zookeeper.ZKUtil(162): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:40,035 DEBUG [RS:1;jenkins-hbase4:36267] zookeeper.ZKUtil(162): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:40,035 DEBUG [RS:2;jenkins-hbase4:33597] zookeeper.ZKUtil(162): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:40,036 DEBUG [RS:0;jenkins-hbase4:35417] zookeeper.ZKUtil(162): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:40,046 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-22 14:53:40,049 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-22 14:53:40,049 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-22 14:53:40,049 DEBUG [RS:2;jenkins-hbase4:33597] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-22 14:53:40,060 INFO [RS:0;jenkins-hbase4:35417] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-22 14:53:40,060 INFO [RS:1;jenkins-hbase4:36267] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-22 14:53:40,060 INFO [RS:2;jenkins-hbase4:33597] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-22 14:53:40,064 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-22 14:53:40,066 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-22 14:53:40,083 INFO [RS:1;jenkins-hbase4:36267] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-22 14:53:40,083 INFO [RS:2;jenkins-hbase4:33597] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-22 14:53:40,083 INFO [RS:0;jenkins-hbase4:35417] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-22 14:53:40,087 INFO [RS:0;jenkins-hbase4:35417] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-22 14:53:40,087 INFO [RS:0;jenkins-hbase4:35417] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,088 INFO [RS:1;jenkins-hbase4:36267] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-22 14:53:40,088 INFO [RS:2;jenkins-hbase4:33597] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-22 14:53:40,089 INFO [RS:1;jenkins-hbase4:36267] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,089 INFO [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-22 14:53:40,089 INFO [RS:2;jenkins-hbase4:33597] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,089 INFO [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-22 14:53:40,090 INFO [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-22 14:53:40,098 INFO [RS:0;jenkins-hbase4:35417] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,098 INFO [RS:2;jenkins-hbase4:33597] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,098 INFO [RS:1;jenkins-hbase4:36267] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,098 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,098 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,098 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,098 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,098 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-22 14:53:40,099 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-22 14:53:40,098 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:0;jenkins-hbase4:35417] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,099 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,100 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,100 DEBUG [RS:1;jenkins-hbase4:36267] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,100 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,100 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-22 14:53:40,100 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,101 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,101 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,101 DEBUG [RS:2;jenkins-hbase4:33597] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-22 14:53:40,104 INFO [RS:0;jenkins-hbase4:35417] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,104 INFO [RS:1;jenkins-hbase4:36267] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,104 INFO [RS:0;jenkins-hbase4:35417] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,104 INFO [RS:1;jenkins-hbase4:36267] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,105 INFO [RS:0;jenkins-hbase4:35417] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,105 INFO [RS:2;jenkins-hbase4:33597] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,105 INFO [RS:1;jenkins-hbase4:36267] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,105 INFO [RS:2;jenkins-hbase4:33597] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,105 INFO [RS:2;jenkins-hbase4:33597] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,123 INFO [RS:1;jenkins-hbase4:36267] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-22 14:53:40,123 INFO [RS:0;jenkins-hbase4:35417] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-22 14:53:40,123 INFO [RS:2;jenkins-hbase4:33597] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-22 14:53:40,129 INFO [RS:2;jenkins-hbase4:33597] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33597,1682175218631-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,129 INFO [RS:1;jenkins-hbase4:36267] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,36267,1682175218583-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,129 INFO [RS:0;jenkins-hbase4:35417] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35417,1682175218529-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,151 INFO [RS:0;jenkins-hbase4:35417] regionserver.Replication(203): jenkins-hbase4.apache.org,35417,1682175218529 started 2023-04-22 14:53:40,151 INFO [RS:2;jenkins-hbase4:33597] regionserver.Replication(203): jenkins-hbase4.apache.org,33597,1682175218631 started 2023-04-22 14:53:40,151 INFO [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35417,1682175218529, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35417, sessionid=0x101ac5cb4e00001 2023-04-22 14:53:40,151 INFO [RS:1;jenkins-hbase4:36267] regionserver.Replication(203): jenkins-hbase4.apache.org,36267,1682175218583 started 2023-04-22 14:53:40,151 INFO [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,33597,1682175218631, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:33597, sessionid=0x101ac5cb4e00003 2023-04-22 14:53:40,151 INFO [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,36267,1682175218583, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:36267, sessionid=0x101ac5cb4e00002 2023-04-22 14:53:40,151 DEBUG [RS:0;jenkins-hbase4:35417] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-22 14:53:40,151 DEBUG [RS:1;jenkins-hbase4:36267] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-22 14:53:40,151 DEBUG [RS:0;jenkins-hbase4:35417] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:40,151 DEBUG [RS:2;jenkins-hbase4:33597] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-22 14:53:40,152 DEBUG [RS:0;jenkins-hbase4:35417] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35417,1682175218529' 2023-04-22 14:53:40,151 DEBUG [RS:1;jenkins-hbase4:36267] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:40,152 DEBUG [RS:0;jenkins-hbase4:35417] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:40,152 DEBUG [RS:2;jenkins-hbase4:33597] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:40,153 DEBUG [RS:2;jenkins-hbase4:33597] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33597,1682175218631' 2023-04-22 14:53:40,153 DEBUG [RS:2;jenkins-hbase4:33597] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:40,152 DEBUG [RS:1;jenkins-hbase4:36267] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36267,1682175218583' 2023-04-22 14:53:40,153 DEBUG [RS:1;jenkins-hbase4:36267] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:40,154 DEBUG [RS:0;jenkins-hbase4:35417] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:40,154 DEBUG [RS:2;jenkins-hbase4:33597] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:40,154 DEBUG [RS:1;jenkins-hbase4:36267] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:40,154 DEBUG [RS:0;jenkins-hbase4:35417] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-22 14:53:40,154 DEBUG [RS:2;jenkins-hbase4:33597] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-22 14:53:40,154 DEBUG [RS:0;jenkins-hbase4:35417] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-22 14:53:40,154 DEBUG [RS:1;jenkins-hbase4:36267] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-22 14:53:40,154 DEBUG [RS:0;jenkins-hbase4:35417] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:40,155 DEBUG [RS:0;jenkins-hbase4:35417] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35417,1682175218529' 2023-04-22 14:53:40,155 DEBUG [RS:0;jenkins-hbase4:35417] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-22 14:53:40,154 DEBUG [RS:2;jenkins-hbase4:33597] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-22 14:53:40,155 DEBUG [RS:1;jenkins-hbase4:36267] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-22 14:53:40,155 DEBUG [RS:2;jenkins-hbase4:33597] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:40,155 DEBUG [RS:1;jenkins-hbase4:36267] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:40,155 DEBUG [RS:2;jenkins-hbase4:33597] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,33597,1682175218631' 2023-04-22 14:53:40,155 DEBUG [RS:2;jenkins-hbase4:33597] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-22 14:53:40,155 DEBUG [RS:1;jenkins-hbase4:36267] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,36267,1682175218583' 2023-04-22 14:53:40,156 DEBUG [RS:0;jenkins-hbase4:35417] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-22 14:53:40,156 DEBUG [RS:1;jenkins-hbase4:36267] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-22 14:53:40,156 DEBUG [RS:2;jenkins-hbase4:33597] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-22 14:53:40,156 DEBUG [RS:0;jenkins-hbase4:35417] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-22 14:53:40,156 DEBUG [RS:1;jenkins-hbase4:36267] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-22 14:53:40,156 INFO [RS:0;jenkins-hbase4:35417] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-22 14:53:40,157 INFO [RS:0;jenkins-hbase4:35417] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-22 14:53:40,157 DEBUG [RS:2;jenkins-hbase4:33597] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-22 14:53:40,157 INFO [RS:2;jenkins-hbase4:33597] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-22 14:53:40,158 INFO [RS:2;jenkins-hbase4:33597] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-22 14:53:40,158 DEBUG [RS:1;jenkins-hbase4:36267] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-22 14:53:40,158 INFO [RS:1;jenkins-hbase4:36267] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-22 14:53:40,158 INFO [RS:1;jenkins-hbase4:36267] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-22 14:53:40,218 DEBUG [jenkins-hbase4:43497] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-04-22 14:53:40,222 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:53:40,228 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:53:40,228 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:53:40,229 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:53:40,229 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:53:40,232 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35417,1682175218529, state=OPENING 2023-04-22 14:53:40,239 DEBUG [PEWorker-1] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-22 14:53:40,241 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:40,241 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-22 14:53:40,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:53:40,271 INFO [RS:2;jenkins-hbase4:33597] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33597%2C1682175218631, suffix=, logDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,33597,1682175218631, archiveDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/oldWALs, maxLogs=32 2023-04-22 14:53:40,271 INFO [RS:0;jenkins-hbase4:35417] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35417%2C1682175218529, suffix=, logDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,35417,1682175218529, archiveDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/oldWALs, maxLogs=32 2023-04-22 14:53:40,280 INFO [RS:1;jenkins-hbase4:36267] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C36267%2C1682175218583, suffix=, logDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,36267,1682175218583, archiveDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/oldWALs, maxLogs=32 2023-04-22 14:53:40,307 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK] 2023-04-22 14:53:40,307 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK] 2023-04-22 14:53:40,307 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK] 2023-04-22 14:53:40,316 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK] 2023-04-22 14:53:40,317 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK] 2023-04-22 14:53:40,317 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK] 2023-04-22 14:53:40,320 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK] 2023-04-22 14:53:40,326 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK] 2023-04-22 14:53:40,326 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK] 2023-04-22 14:53:40,334 INFO [RS:0;jenkins-hbase4:35417] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,35417,1682175218529/jenkins-hbase4.apache.org%2C35417%2C1682175218529.1682175220276 2023-04-22 14:53:40,335 DEBUG [RS:0;jenkins-hbase4:35417] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK], DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK], DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK]] 2023-04-22 14:53:40,339 INFO [RS:2;jenkins-hbase4:33597] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,33597,1682175218631/jenkins-hbase4.apache.org%2C33597%2C1682175218631.1682175220276 2023-04-22 14:53:40,340 DEBUG [RS:2;jenkins-hbase4:33597] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK], DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK], DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK]] 2023-04-22 14:53:40,341 INFO [RS:1;jenkins-hbase4:36267] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,36267,1682175218583/jenkins-hbase4.apache.org%2C36267%2C1682175218583.1682175220282 2023-04-22 14:53:40,342 DEBUG [RS:1;jenkins-hbase4:36267] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK], DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK], DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK]] 2023-04-22 14:53:40,436 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:40,439 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-22 14:53:40,444 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44470, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-22 14:53:40,457 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-22 14:53:40,457 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-22 14:53:40,461 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35417%2C1682175218529.meta, suffix=.meta, logDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,35417,1682175218529, archiveDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/oldWALs, maxLogs=32 2023-04-22 14:53:40,487 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK] 2023-04-22 14:53:40,488 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK] 2023-04-22 14:53:40,490 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK] 2023-04-22 14:53:40,495 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,35417,1682175218529/jenkins-hbase4.apache.org%2C35417%2C1682175218529.meta.1682175220463.meta 2023-04-22 14:53:40,496 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:41961,DS-e611b5f6-8d5c-4d26-948f-d907b53a22e6,DISK], DatanodeInfoWithStorage[127.0.0.1:35001,DS-abe26421-c7b5-427b-b84d-ea5d85a0cd7c,DISK], DatanodeInfoWithStorage[127.0.0.1:42675,DS-8329503a-6d5e-45e3-9091-b3a05c845e79,DISK]] 2023-04-22 14:53:40,496 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:40,501 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:53:40,502 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-22 14:53:40,520 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-22 14:53:40,523 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-22 14:53:40,528 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-22 14:53:40,528 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:40,528 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-22 14:53:40,528 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-22 14:53:40,532 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-22 14:53:40,534 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/info 2023-04-22 14:53:40,534 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/info 2023-04-22 14:53:40,535 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-22 14:53:40,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:40,536 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-22 14:53:40,537 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/rep_barrier 2023-04-22 14:53:40,537 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/rep_barrier 2023-04-22 14:53:40,538 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-22 14:53:40,539 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:40,539 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-22 14:53:40,541 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/table 2023-04-22 14:53:40,541 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/table 2023-04-22 14:53:40,541 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-22 14:53:40,542 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:40,544 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740 2023-04-22 14:53:40,548 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740 2023-04-22 14:53:40,552 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-04-22 14:53:40,555 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-22 14:53:40,556 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=58958246, jitterRate=-0.1214536726474762}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-04-22 14:53:40,557 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-22 14:53:40,570 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1682175220424 2023-04-22 14:53:40,596 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35417,1682175218529, state=OPEN 2023-04-22 14:53:40,600 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-22 14:53:40,601 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-22 14:53:40,602 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-22 14:53:40,602 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-22 14:53:40,607 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-22 14:53:40,608 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35417,1682175218529 in 356 msec 2023-04-22 14:53:40,615 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-22 14:53:40,615 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 565 msec 2023-04-22 14:53:40,622 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 848 msec 2023-04-22 14:53:40,623 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1682175220623, completionTime=-1 2023-04-22 14:53:40,623 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-04-22 14:53:40,623 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:40,700 DEBUG [hconnection-0x42387a87-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:40,704 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44480, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:40,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:40,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175280723 2023-04-22 14:53:40,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175340723 2023-04-22 14:53:40,723 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 100 msec 2023-04-22 14:53:40,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43497,1682175217266-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43497,1682175217266-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,749 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43497,1682175217266-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,751 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:43497, period=300000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,751 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-22 14:53:40,762 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-22 14:53:40,770 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-22 14:53:40,772 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-22 14:53:40,784 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:40,787 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:53:40,789 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:53:40,814 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:53:40,817 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65 empty. 2023-04-22 14:53:40,817 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:53:40,818 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-22 14:53:40,857 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:40,859 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => d65d90b6fbc4a32d4609259a87039a65, NAME => 'hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:53:40,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:40,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing d65d90b6fbc4a32d4609259a87039a65, disabling compactions & flushes 2023-04-22 14:53:40,879 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:53:40,879 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:53:40,880 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. after waiting 0 ms 2023-04-22 14:53:40,880 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:53:40,880 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:53:40,880 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for d65d90b6fbc4a32d4609259a87039a65: 2023-04-22 14:53:40,886 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:53:40,904 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682175220889"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175220889"}]},"ts":"1682175220889"} 2023-04-22 14:53:40,933 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:53:40,935 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:53:40,940 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175220935"}]},"ts":"1682175220935"} 2023-04-22 14:53:40,946 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-22 14:53:40,954 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:53:40,955 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:53:40,955 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:53:40,955 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:53:40,955 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:53:40,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d65d90b6fbc4a32d4609259a87039a65, ASSIGN}] 2023-04-22 14:53:40,962 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=d65d90b6fbc4a32d4609259a87039a65, ASSIGN 2023-04-22 14:53:40,964 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=d65d90b6fbc4a32d4609259a87039a65, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:53:41,116 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:53:41,117 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d65d90b6fbc4a32d4609259a87039a65, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:41,118 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682175221117"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175221117"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175221117"}]},"ts":"1682175221117"} 2023-04-22 14:53:41,123 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure d65d90b6fbc4a32d4609259a87039a65, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:41,277 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:41,278 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-22 14:53:41,283 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50198, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-22 14:53:41,289 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:53:41,291 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d65d90b6fbc4a32d4609259a87039a65, NAME => 'hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:41,292 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:53:41,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:53:41,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:41,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:53:41,293 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:53:41,296 INFO [StoreOpener-d65d90b6fbc4a32d4609259a87039a65-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:53:41,298 DEBUG [StoreOpener-d65d90b6fbc4a32d4609259a87039a65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65/info 2023-04-22 14:53:41,298 DEBUG [StoreOpener-d65d90b6fbc4a32d4609259a87039a65-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65/info 2023-04-22 14:53:41,299 INFO [StoreOpener-d65d90b6fbc4a32d4609259a87039a65-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d65d90b6fbc4a32d4609259a87039a65 columnFamilyName info 2023-04-22 14:53:41,300 INFO [StoreOpener-d65d90b6fbc4a32d4609259a87039a65-1] regionserver.HStore(310): Store=d65d90b6fbc4a32d4609259a87039a65/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:41,302 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:53:41,303 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:53:41,307 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:53:41,311 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:41,312 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d65d90b6fbc4a32d4609259a87039a65; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72750290, jitterRate=0.08406379818916321}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-22 14:53:41,312 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d65d90b6fbc4a32d4609259a87039a65: 2023-04-22 14:53:41,314 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65., pid=6, masterSystemTime=1682175221277 2023-04-22 14:53:41,319 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:53:41,320 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:53:41,321 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:41,322 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1682175221321"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175221321"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175221321"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175221321"}]},"ts":"1682175221321"} 2023-04-22 14:53:41,331 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-22 14:53:41,331 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure d65d90b6fbc4a32d4609259a87039a65, server=jenkins-hbase4.apache.org,36267,1682175218583 in 203 msec 2023-04-22 14:53:41,335 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-22 14:53:41,336 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=d65d90b6fbc4a32d4609259a87039a65, ASSIGN in 373 msec 2023-04-22 14:53:41,337 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:53:41,338 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175221338"}]},"ts":"1682175221338"} 2023-04-22 14:53:41,341 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-22 14:53:41,346 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:53:41,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 573 msec 2023-04-22 14:53:41,387 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-22 14:53:41,390 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-22 14:53:41,390 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:41,413 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:41,417 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:41,431 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:41,451 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-22 14:53:41,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-04-22 14:53:41,465 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:41,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-22 14:53:41,486 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 20 msec 2023-04-22 14:53:41,502 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-22 14:53:41,505 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-22 14:53:41,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 2.803sec 2023-04-22 14:53:41,509 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-22 14:53:41,510 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-22 14:53:41,511 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-22 14:53:41,512 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43497,1682175217266-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-22 14:53:41,513 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43497,1682175217266-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-22 14:53:41,523 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-22 14:53:41,568 DEBUG [Listener at localhost/43403] zookeeper.ReadOnlyZKClient(139): Connect 0x3e0ba461 to 127.0.0.1:50713 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-22 14:53:41,573 DEBUG [Listener at localhost/43403] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2dead8c8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-22 14:53:41,588 DEBUG [hconnection-0x6454e058-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:41,632 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44486, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:41,642 INFO [Listener at localhost/43403] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:53:41,672 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=360, OpenFileDescriptor=692, MaxFileDescriptor=60000, SystemLoadAverage=316, ProcessCount=172, AvailableMemoryMB=13284 2023-04-22 14:53:41,677 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:53:41,685 DEBUG [Listener at localhost/43403] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-22 14:53:41,689 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51926, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-22 14:53:41,702 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:53:41,703 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:53:41,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:53:41,709 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:41,711 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:53:41,713 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:53:41,716 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:41,717 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099 empty. 2023-04-22 14:53:41,719 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:41,719 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-04-22 14:53:41,741 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:41,743 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(7675): creating {ENCODED => 1539dac144b8acabc3b8d8b035e2f099, NAME => 'testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionUnevenDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:53:41,764 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:41,764 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1604): Closing 1539dac144b8acabc3b8d8b035e2f099, disabling compactions & flushes 2023-04-22 14:53:41,764 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:41,764 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:41,764 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. after waiting 0 ms 2023-04-22 14:53:41,764 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:41,764 INFO [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:41,764 DEBUG [RegionOpenAndInit-testSplitTableRegionUnevenDaughter-pool-0] regionserver.HRegion(1558): Region close journal for 1539dac144b8acabc3b8d8b035e2f099: 2023-04-22 14:53:41,768 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:53:41,770 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1682175221770"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175221770"}]},"ts":"1682175221770"} 2023-04-22 14:53:41,773 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:53:41,774 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:53:41,775 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175221775"}]},"ts":"1682175221775"} 2023-04-22 14:53:41,777 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLING in hbase:meta 2023-04-22 14:53:41,781 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:53:41,783 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:53:41,783 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:53:41,783 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:53:41,783 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:53:41,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1539dac144b8acabc3b8d8b035e2f099, ASSIGN}] 2023-04-22 14:53:41,787 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1539dac144b8acabc3b8d8b035e2f099, ASSIGN 2023-04-22 14:53:41,788 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1539dac144b8acabc3b8d8b035e2f099, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:53:41,938 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:53:41,939 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1539dac144b8acabc3b8d8b035e2f099, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:41,940 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1682175221939"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175221939"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175221939"}]},"ts":"1682175221939"} 2023-04-22 14:53:41,943 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 1539dac144b8acabc3b8d8b035e2f099, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:42,103 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:42,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1539dac144b8acabc3b8d8b035e2f099, NAME => 'testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:42,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:53:42,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:42,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:42,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:42,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:42,106 INFO [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:42,108 DEBUG [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1 2023-04-22 14:53:42,108 DEBUG [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1 2023-04-22 14:53:42,109 INFO [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1539dac144b8acabc3b8d8b035e2f099 columnFamilyName cf1 2023-04-22 14:53:42,110 INFO [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] regionserver.HStore(310): Store=1539dac144b8acabc3b8d8b035e2f099/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:42,110 INFO [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:42,112 DEBUG [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2 2023-04-22 14:53:42,112 DEBUG [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2 2023-04-22 14:53:42,112 INFO [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1539dac144b8acabc3b8d8b035e2f099 columnFamilyName cf2 2023-04-22 14:53:42,113 INFO [StoreOpener-1539dac144b8acabc3b8d8b035e2f099-1] regionserver.HStore(310): Store=1539dac144b8acabc3b8d8b035e2f099/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:42,115 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:42,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:42,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:53:42,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:42,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:42,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1539dac144b8acabc3b8d8b035e2f099; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67911563, jitterRate=0.01196114718914032}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:53:42,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1539dac144b8acabc3b8d8b035e2f099: 2023-04-22 14:53:42,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099., pid=11, masterSystemTime=1682175222096 2023-04-22 14:53:42,129 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:42,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:42,130 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=1539dac144b8acabc3b8d8b035e2f099, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,131 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1682175222130"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175222130"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175222130"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175222130"}]},"ts":"1682175222130"} 2023-04-22 14:53:42,137 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-22 14:53:42,138 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 1539dac144b8acabc3b8d8b035e2f099, server=jenkins-hbase4.apache.org,36267,1682175218583 in 191 msec 2023-04-22 14:53:42,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-22 14:53:42,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1539dac144b8acabc3b8d8b035e2f099, ASSIGN in 355 msec 2023-04-22 14:53:42,142 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:53:42,143 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175222143"}]},"ts":"1682175222143"} 2023-04-22 14:53:42,145 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=ENABLED in hbase:meta 2023-04-22 14:53:42,148 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionUnevenDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:53:42,150 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter in 441 msec 2023-04-22 14:53:42,214 DEBUG [Listener at localhost/43403] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:53:42,218 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:53:42,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:42,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,246 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:42,247 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,247 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-22 14:53:42,247 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-22 14:53:42,248 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,248 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,251 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,251 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,251 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,256 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:42,257 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:42,257 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:42,255 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,257 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:42,257 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:42,257 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:42,257 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:42,258 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:42,258 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:42,258 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:42,259 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. started... 2023-04-22 14:53:42,260 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1539dac144b8acabc3b8d8b035e2f099 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-22 14:53:42,260 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,260 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,260 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,260 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,260 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:42,260 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:42,335 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/5a63098d55b14793a64159229531804f 2023-04-22 14:53:42,406 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/5d7f1432245b4bb19d5ef86844457849 2023-04-22 14:53:42,420 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/5a63098d55b14793a64159229531804f as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f 2023-04-22 14:53:42,429 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:53:42,432 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/5d7f1432245b4bb19d5ef86844457849 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849 2023-04-22 14:53:42,442 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:53:42,444 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 1539dac144b8acabc3b8d8b035e2f099 in 184ms, sequenceid=6, compaction requested=false 2023-04-22 14:53:42,446 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:42,448 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1539dac144b8acabc3b8d8b035e2f099: 2023-04-22 14:53:42,448 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:42,449 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:42,449 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:42,449 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,449 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:42,449 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:42,453 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,453 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:42,453 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,453 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:42,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-22 14:53:42,456 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-22 14:53:42,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@245c9a51[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:42,456 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,462 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:42,462 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:42,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:42,462 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:42,466 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:42,466 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:42,466 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:42,467 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:42,467 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:42,467 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:42,467 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,467 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:42,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:42,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:42,467 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:42,468 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:42,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:42,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:42,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:42,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:42,473 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-22 14:53:42,473 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:42,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:53:42,473 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:42,474 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,474 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:42,477 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,477 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,477 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,477 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,477 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,478 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:42,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,478 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,478 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:42,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,479 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,480 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:42,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,482 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:42,483 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,483 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:42,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:42,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,496 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:42,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,497 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,498 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,498 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,499 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:42,499 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:42,599 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:42,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:42,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:42,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:42,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,635 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:42,635 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,636 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-22 14:53:42,636 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-22 14:53:42,637 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,637 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,638 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,638 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,638 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,638 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,639 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,639 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:42,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:42,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:42,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,641 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:42,641 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:42,641 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,641 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,643 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:42,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:42,643 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:42,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,643 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:42,644 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:42,644 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:42,644 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,643 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:42,644 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:42,644 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:42,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:42,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:42,645 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:42,645 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. started... 2023-04-22 14:53:42,646 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1539dac144b8acabc3b8d8b035e2f099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:42,647 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,649 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,649 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,649 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:42,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:42,669 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/88655a01a6794c30a88fb4b3f8d5dc2f 2023-04-22 14:53:42,695 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/558446f99181496dbf918ba699920f52 2023-04-22 14:53:42,706 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/88655a01a6794c30a88fb4b3f8d5dc2f as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f 2023-04-22 14:53:42,716 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:53:42,718 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/558446f99181496dbf918ba699920f52 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52 2023-04-22 14:53:42,729 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:53:42,731 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1539dac144b8acabc3b8d8b035e2f099 in 85ms, sequenceid=19, compaction requested=false 2023-04-22 14:53:42,731 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1539dac144b8acabc3b8d8b035e2f099: 2023-04-22 14:53:42,731 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:42,731 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:42,732 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:42,732 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,732 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:42,732 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:42,734 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,734 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:42,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:42,734 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,735 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:42,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:42,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:42,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:42,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-22 14:53:42,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f51361e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:42,738 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-22 14:53:42,738 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,739 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,739 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,740 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:42,740 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:42,740 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:42,740 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,739 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,740 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,740 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:42,740 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:42,740 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:42,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:42,741 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:42,743 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:42,743 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:42,743 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:42,745 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:42,745 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:42,745 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:42,746 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:42,746 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,746 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:42,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:42,746 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:42,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:42,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:42,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:42,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:53:42,757 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-22 14:53:42,757 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:42,757 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:42,757 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,757 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:42,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,762 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:42,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:42,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,764 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,765 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:42,766 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:42,770 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,771 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,771 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,783 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,783 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:42,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,783 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,783 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,783 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:42,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:42,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:42,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,784 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,784 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,784 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,785 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:42,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:42,785 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,785 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,785 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,785 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,785 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,785 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,885 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:42,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:42,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:42,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:42,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,911 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:42,912 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,912 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-22 14:53:42,912 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-22 14:53:42,913 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,913 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,915 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,915 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,915 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,915 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:42,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,915 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,915 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:42,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:42,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,917 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:42,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:42,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,922 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:42,924 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:42,923 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:42,924 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. started... 2023-04-22 14:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:42,924 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1539dac144b8acabc3b8d8b035e2f099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:42,927 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,929 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,929 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,929 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:42,929 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,929 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:42,945 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/11b157fcfe7349bca3bc71853427cb50 2023-04-22 14:53:42,968 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/2178a13bef814adc82ef58ae3f82fc7a 2023-04-22 14:53:42,977 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/11b157fcfe7349bca3bc71853427cb50 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50 2023-04-22 14:53:42,985 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:53:42,986 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/2178a13bef814adc82ef58ae3f82fc7a as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a 2023-04-22 14:53:42,993 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:53:42,994 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1539dac144b8acabc3b8d8b035e2f099 in 70ms, sequenceid=32, compaction requested=true 2023-04-22 14:53:42,994 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1539dac144b8acabc3b8d8b035e2f099: 2023-04-22 14:53:42,994 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:42,994 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:42,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:42,997 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,997 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:42,998 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,998 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-22 14:53:43,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bb88206[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:43,000 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-22 14:53:43,000 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,001 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,001 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,001 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,001 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,001 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:43,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:43,001 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,002 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:43,005 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,005 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,005 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,007 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,007 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,007 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,007 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,007 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,008 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,008 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:43,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:53:43,013 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-22 14:53:43,013 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:43,013 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:43,013 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,013 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:43,015 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,015 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,015 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,015 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,015 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,016 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,015 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,016 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,016 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,017 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,019 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,019 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,020 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,024 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,024 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,024 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,024 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,024 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,025 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,024 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,024 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,025 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:43,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,024 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,024 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,025 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,025 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,025 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:43,026 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,026 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:43,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,026 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,026 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,026 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,026 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,126 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:43,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:43,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:43,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:43,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,153 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:43,153 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,154 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-22 14:53:43,154 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-22 14:53:43,154 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,154 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,156 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,156 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,156 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,156 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,156 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,157 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,158 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:43,158 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,161 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:43,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:43,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:43,162 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:43,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,162 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:43,162 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:43,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,162 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,162 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:43,163 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:43,163 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:43,163 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:43,163 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,163 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:43,164 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:43,164 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:43,164 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:43,164 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. started... 2023-04-22 14:53:43,164 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1539dac144b8acabc3b8d8b035e2f099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:43,165 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,167 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:43,188 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/1f5011b82cc645eba7885b8f9d4aceb5 2023-04-22 14:53:43,213 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/cb259bd4795f45b18d809f71f95e1610 2023-04-22 14:53:43,222 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/1f5011b82cc645eba7885b8f9d4aceb5 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5 2023-04-22 14:53:43,230 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:53:43,231 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/cb259bd4795f45b18d809f71f95e1610 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610 2023-04-22 14:53:43,238 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:53:43,239 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1539dac144b8acabc3b8d8b035e2f099 in 75ms, sequenceid=45, compaction requested=true 2023-04-22 14:53:43,239 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1539dac144b8acabc3b8d8b035e2f099: 2023-04-22 14:53:43,239 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:43,240 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:43,240 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:43,240 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,240 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:43,241 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:43,244 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,244 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,245 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:43,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-22 14:53:43,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a942e99[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:43,247 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-22 14:53:43,247 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,248 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,248 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,248 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,248 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,248 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,248 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,249 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:43,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:43,249 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:43,250 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,250 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,250 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,251 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,253 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,254 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:43,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:53:43,258 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-22 14:53:43,258 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:43,258 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:43,258 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,258 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:43,260 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,260 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,260 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,260 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,260 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,260 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,260 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,261 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,262 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,262 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,264 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,265 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,265 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,268 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,268 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,268 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,268 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,268 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,268 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,268 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,268 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,269 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:43,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter because node does not exist (not an error) 2023-04-22 14:53:43,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,268 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:43,269 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,269 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,270 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,270 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,270 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,270 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:43,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,270 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:43,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,270 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,271 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,271 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,371 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:43,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:43,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:43,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:43,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,395 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:43,395 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,396 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-22 14:53:43,396 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-22 14:53:43,397 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,397 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,398 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,398 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,398 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,398 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,399 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,399 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:43,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:43,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:43,400 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,400 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:43,402 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,402 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:43,404 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:43,404 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:43,404 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:43,404 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:43,404 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:43,404 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. started... 2023-04-22 14:53:43,404 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:43,404 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1539dac144b8acabc3b8d8b035e2f099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:43,406 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,407 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,407 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,407 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:43,407 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,407 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:43,422 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/ae6318f1ccfb421eaebbac78986da928 2023-04-22 14:53:43,451 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/29414784ccf94b5396b1f647a1defac5 2023-04-22 14:53:43,459 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/ae6318f1ccfb421eaebbac78986da928 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/ae6318f1ccfb421eaebbac78986da928 2023-04-22 14:53:43,466 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/ae6318f1ccfb421eaebbac78986da928, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:53:43,467 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/29414784ccf94b5396b1f647a1defac5 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/29414784ccf94b5396b1f647a1defac5 2023-04-22 14:53:43,475 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/29414784ccf94b5396b1f647a1defac5, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:53:43,476 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1539dac144b8acabc3b8d8b035e2f099 in 72ms, sequenceid=58, compaction requested=true 2023-04-22 14:53:43,476 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1539dac144b8acabc3b8d8b035e2f099: 2023-04-22 14:53:43,476 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:43,476 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:43,478 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:43,478 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,478 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:43,478 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:43,480 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,480 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-22 14:53:43,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1adaa988[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:43,483 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-22 14:53:43,483 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,484 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,484 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,484 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,485 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,485 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:43,487 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,487 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,487 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,489 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:43,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:53:43,493 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-22 14:53:43,493 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:43,493 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:43,493 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,493 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:43,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,496 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,496 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,497 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,498 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,498 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,499 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,505 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,505 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,505 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,506 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,505 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,505 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,506 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,506 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,506 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,507 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,507 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,506 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:43,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:43,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,507 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,508 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:43,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,508 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,507 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,508 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,508 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,508 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,508 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,608 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:43,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:43,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:43,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionUnevenDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:43,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,632 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:43,632 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,633 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' 2023-04-22 14:53:43,633 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionUnevenDaughter', kicking off acquire phase on members. 2023-04-22 14:53:43,634 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,634 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,635 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,635 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,635 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,635 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,636 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,636 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:43,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:43,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:43,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionUnevenDaughter is still present, removing 2023-04-22 14:53:43,638 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:43,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,638 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,638 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:43,639 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:43,639 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,640 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:43,640 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionUnevenDaughter' with timeout 60000ms 2023-04-22 14:53:43,640 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:43,639 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:43,640 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:43,640 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:43,640 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:43,640 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionUnevenDaughter' starting 'acquire' stage 2023-04-22 14:53:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:43,641 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:43,641 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. started... 2023-04-22 14:53:43,641 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 1539dac144b8acabc3b8d8b035e2f099 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:43,642 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,642 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,643 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,643 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:43,643 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,643 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:43,667 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/3a4b982ef80e4b77a6f2a83e7ac1f59f 2023-04-22 14:53:43,705 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/7b6e534a381843d285e051b58fac6eed 2023-04-22 14:53:43,715 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/3a4b982ef80e4b77a6f2a83e7ac1f59f as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/3a4b982ef80e4b77a6f2a83e7ac1f59f 2023-04-22 14:53:43,723 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/3a4b982ef80e4b77a6f2a83e7ac1f59f, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:53:43,724 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/7b6e534a381843d285e051b58fac6eed as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/7b6e534a381843d285e051b58fac6eed 2023-04-22 14:53:43,733 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/7b6e534a381843d285e051b58fac6eed, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:53:43,735 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 1539dac144b8acabc3b8d8b035e2f099 in 93ms, sequenceid=71, compaction requested=true 2023-04-22 14:53:43,735 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 1539dac144b8acabc3b8d8b035e2f099: 2023-04-22 14:53:43,735 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionUnevenDaughter' locally acquired 2023-04-22 14:53:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionUnevenDaughter) in zk 2023-04-22 14:53:43,738 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,738 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,738 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,738 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionUnevenDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegionUnevenDaughter' on coordinator 2023-04-22 14:53:43,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ccd8d0b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:43,741 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionUnevenDaughter' starting 'in-barrier' execution. 2023-04-22 14:53:43,741 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,743 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,744 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,744 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,744 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:43,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,744 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:43,744 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,744 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,744 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:43,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,744 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionUnevenDaughter' received 'reached' from coordinator. 2023-04-22 14:53:43,744 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionUnevenDaughter' locally completed 2023-04-22 14:53:43,744 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionUnevenDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:43,745 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,745 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,745 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,745 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,746 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,747 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionUnevenDaughter' has notified controller of completion 2023-04-22 14:53:43,747 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,747 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionUnevenDaughter' completed. 2023-04-22 14:53:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionUnevenDaughter' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:43,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegionUnevenDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:53:43,751 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionUnevenDaughter' execution completed 2023-04-22 14:53:43,751 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:43,751 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:43,751 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,751 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionUnevenDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:43,752 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,752 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,752 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,752 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,752 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,753 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,752 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:43,753 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:43,754 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,754 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,758 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:43,758 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,758 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,762 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,762 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,762 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,762 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,762 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,762 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionUnevenDaughter' 2023-04-22 14:53:43,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:43,762 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:43,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,762 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:43,763 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:43,763 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,763 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,763 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:43,763 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,763 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:43,763 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,763 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,763 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionUnevenDaughter 2023-04-22 14:53:43,763 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:43,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,764 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionUnevenDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:43,765 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:43,865 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:43,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:43,902 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,911 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:43,912 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:43,912 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:43,913 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:43,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1539dac144b8acabc3b8d8b035e2f099, UNASSIGN}] 2023-04-22 14:53:43,931 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1539dac144b8acabc3b8d8b035e2f099, UNASSIGN 2023-04-22 14:53:43,933 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1539dac144b8acabc3b8d8b035e2f099, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:43,933 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1682175223933"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175223933"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175223933"}]},"ts":"1682175223933"} 2023-04-22 14:53:43,937 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 1539dac144b8acabc3b8d8b035e2f099, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:44,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1539dac144b8acabc3b8d8b035e2f099, disabling compactions & flushes 2023-04-22 14:53:44,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. after waiting 0 ms 2023-04-22 14:53:44,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:44,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1539dac144b8acabc3b8d8b035e2f099 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-22 14:53:44,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/f4413a3d6e6f41328ec1e64be494fe81 2023-04-22 14:53:44,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/fc94602f35ab4569817df2912b29b06e 2023-04-22 14:53:44,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf1/f4413a3d6e6f41328ec1e64be494fe81 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/f4413a3d6e6f41328ec1e64be494fe81 2023-04-22 14:53:44,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/f4413a3d6e6f41328ec1e64be494fe81, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:53:44,303 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.tmp/cf2/fc94602f35ab4569817df2912b29b06e as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/fc94602f35ab4569817df2912b29b06e 2023-04-22 14:53:44,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/fc94602f35ab4569817df2912b29b06e, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:53:44,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 1539dac144b8acabc3b8d8b035e2f099 in 200ms, sequenceid=82, compaction requested=true 2023-04-22 14:53:44,377 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-22 14:53:44,379 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:53:44,379 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. 2023-04-22 14:53:44,380 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1539dac144b8acabc3b8d8b035e2f099: 2023-04-22 14:53:44,390 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,390 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=1539dac144b8acabc3b8d8b035e2f099, regionState=CLOSED 2023-04-22 14:53:44,392 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1682175224390"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175224390"}]},"ts":"1682175224390"} 2023-04-22 14:53:44,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-22 14:53:44,402 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 1539dac144b8acabc3b8d8b035e2f099, server=jenkins-hbase4.apache.org,36267,1682175218583 in 458 msec 2023-04-22 14:53:44,407 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-22 14:53:44,407 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=1539dac144b8acabc3b8d8b035e2f099, UNASSIGN in 477 msec 2023-04-22 14:53:44,471 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=12 splitting 14 storefiles, region=1539dac144b8acabc3b8d8b035e2f099, threads=14 2023-04-22 14:53:44,476 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,484 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/29414784ccf94b5396b1f647a1defac5 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,492 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,496 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,497 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/7b6e534a381843d285e051b58fac6eed for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,500 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,505 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/fc94602f35ab4569817df2912b29b06e for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,511 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,512 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,516 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/3a4b982ef80e4b77a6f2a83e7ac1f59f for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,520 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,541 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,542 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/ae6318f1ccfb421eaebbac78986da928 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,546 DEBUG [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/29414784ccf94b5396b1f647a1defac5, top=true 2023-04-22 14:53:44,569 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=12 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/f4413a3d6e6f41328ec1e64be494fe81 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,642 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/fc94602f35ab4569817df2912b29b06e, top=true 2023-04-22 14:53:44,655 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/7b6e534a381843d285e051b58fac6eed, top=true 2023-04-22 14:53:44,728 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/f4413a3d6e6f41328ec1e64be494fe81, top=true 2023-04-22 14:53:44,743 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/ae6318f1ccfb421eaebbac78986da928, top=true 2023-04-22 14:53:44,749 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.splits/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-fc94602f35ab4569817df2912b29b06e for child: cbd04a56b211199f6d2f4a000c3bb4f6, parent: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,750 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/fc94602f35ab4569817df2912b29b06e for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,752 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.splits/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-7b6e534a381843d285e051b58fac6eed for child: cbd04a56b211199f6d2f4a000c3bb4f6, parent: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,752 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/7b6e534a381843d285e051b58fac6eed for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,781 INFO [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.splits/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-29414784ccf94b5396b1f647a1defac5 for child: cbd04a56b211199f6d2f4a000c3bb4f6, parent: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,781 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/29414784ccf94b5396b1f647a1defac5 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,829 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.splits/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-ae6318f1ccfb421eaebbac78986da928 for child: cbd04a56b211199f6d2f4a000c3bb4f6, parent: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,829 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/ae6318f1ccfb421eaebbac78986da928 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,830 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.splits/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-f4413a3d6e6f41328ec1e64be494fe81 for child: cbd04a56b211199f6d2f4a000c3bb4f6, parent: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,830 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/f4413a3d6e6f41328ec1e64be494fe81 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,839 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/3a4b982ef80e4b77a6f2a83e7ac1f59f, top=true 2023-04-22 14:53:44,943 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/.splits/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-3a4b982ef80e4b77a6f2a83e7ac1f59f for child: cbd04a56b211199f6d2f4a000c3bb4f6, parent: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:44,943 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/3a4b982ef80e4b77a6f2a83e7ac1f59f for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:45,068 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:45,068 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:45,113 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:45,154 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:45,157 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:45,158 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:45,178 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50 for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:45,600 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=12 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f for region: 1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:45,600 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=12 split storefiles for region 1539dac144b8acabc3b8d8b035e2f099 Daughter A: 8 storefiles, Daughter B: 14 storefiles. 2023-04-22 14:53:45,760 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:53:45,766 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:53:45,770 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1682175225769"},{"qualifier":"splitA","vlen":70,"tag":[],"timestamp":"1682175225769"},{"qualifier":"splitB","vlen":70,"tag":[],"timestamp":"1682175225769"}]},"ts":"1682175225769"} 2023-04-22 14:53:45,770 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175225769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175225769"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175225769"}]},"ts":"1682175225769"} 2023-04-22 14:53:45,770 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175225769"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175225769"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175225769"}]},"ts":"1682175225769"} 2023-04-22 14:53:45,833 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ce3321b32c9468788bb2b99657f8761a, ASSIGN}, {pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cbd04a56b211199f6d2f4a000c3bb4f6, ASSIGN}] 2023-04-22 14:53:45,835 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cbd04a56b211199f6d2f4a000c3bb4f6, ASSIGN 2023-04-22 14:53:45,837 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=16, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cbd04a56b211199f6d2f4a000c3bb4f6, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:53:45,839 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ce3321b32c9468788bb2b99657f8761a, ASSIGN 2023-04-22 14:53:45,840 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ce3321b32c9468788bb2b99657f8761a, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:53:45,988 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-22 14:53:45,990 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=cbd04a56b211199f6d2f4a000c3bb4f6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:45,990 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175225990"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175225990"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175225990"}]},"ts":"1682175225990"} 2023-04-22 14:53:45,993 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE; OpenRegionProcedure cbd04a56b211199f6d2f4a000c3bb4f6, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:45,999 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=ce3321b32c9468788bb2b99657f8761a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:46,000 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175225999"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175225999"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175225999"}]},"ts":"1682175225999"} 2023-04-22 14:53:46,004 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=15, state=RUNNABLE; OpenRegionProcedure ce3321b32c9468788bb2b99657f8761a, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:46,107 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:53:46,155 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:46,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => cbd04a56b211199f6d2f4a000c3bb4f6, NAME => 'testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.', STARTKEY => '26', ENDKEY => ''} 2023-04-22 14:53:46,156 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:53:46,156 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:46,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:46,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:46,157 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:46,187 INFO [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:46,192 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1 2023-04-22 14:53:46,192 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1 2023-04-22 14:53:46,193 INFO [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cbd04a56b211199f6d2f4a000c3bb4f6 columnFamilyName cf1 2023-04-22 14:53:46,288 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-22 14:53:46,289 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-22 14:53:46,305 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50-top 2023-04-22 14:53:46,317 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5-top 2023-04-22 14:53:46,326 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f-top 2023-04-22 14:53:46,335 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f-top 2023-04-22 14:53:46,344 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-3a4b982ef80e4b77a6f2a83e7ac1f59f 2023-04-22 14:53:46,356 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-ae6318f1ccfb421eaebbac78986da928 2023-04-22 14:53:46,364 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-f4413a3d6e6f41328ec1e64be494fe81 2023-04-22 14:53:46,365 INFO [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(310): Store=cbd04a56b211199f6d2f4a000c3bb4f6/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:46,365 INFO [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:46,367 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2 2023-04-22 14:53:46,367 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2 2023-04-22 14:53:46,368 INFO [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region cbd04a56b211199f6d2f4a000c3bb4f6 columnFamilyName cf2 2023-04-22 14:53:46,398 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a-top 2023-04-22 14:53:46,415 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52-top 2023-04-22 14:53:46,429 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849-top 2023-04-22 14:53:46,442 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610-top 2023-04-22 14:53:46,456 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-29414784ccf94b5396b1f647a1defac5 2023-04-22 14:53:46,464 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-7b6e534a381843d285e051b58fac6eed 2023-04-22 14:53:46,479 DEBUG [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-fc94602f35ab4569817df2912b29b06e 2023-04-22 14:53:46,480 INFO [StoreOpener-cbd04a56b211199f6d2f4a000c3bb4f6-1] regionserver.HStore(310): Store=cbd04a56b211199f6d2f4a000c3bb4f6/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:46,481 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:46,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:46,489 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:53:46,491 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:46,493 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened cbd04a56b211199f6d2f4a000c3bb4f6; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59534892, jitterRate=-0.11286097764968872}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:53:46,493 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for cbd04a56b211199f6d2f4a000c3bb4f6: 2023-04-22 14:53:46,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6., pid=17, masterSystemTime=1682175226147 2023-04-22 14:53:46,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:46,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:46,502 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:53:46,502 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:53:46,504 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:46,504 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:46,505 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1912): cbd04a56b211199f6d2f4a000c3bb4f6/cf1 is initiating minor compaction (all files) 2023-04-22 14:53:46,505 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:46,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:46,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:46,506 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=16 updating hbase:meta row=cbd04a56b211199f6d2f4a000c3bb4f6, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:46,504 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1912): cbd04a56b211199f6d2f4a000c3bb4f6/cf2 is initiating minor compaction (all files) 2023-04-22 14:53:46,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ce3321b32c9468788bb2b99657f8761a, NAME => 'testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.', STARTKEY => '', ENDKEY => '26'} 2023-04-22 14:53:46,506 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2259): Starting compaction of cbd04a56b211199f6d2f4a000c3bb4f6/cf2 in testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:46,506 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175226506"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175226506"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175226506"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175226506"}]},"ts":"1682175226506"} 2023-04-22 14:53:46,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:53:46,507 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-29414784ccf94b5396b1f647a1defac5, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-7b6e534a381843d285e051b58fac6eed, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-fc94602f35ab4569817df2912b29b06e] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/.tmp, totalSize=34.9 K 2023-04-22 14:53:46,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionUnevenDaughter ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:46,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:46,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:46,521 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:46,522 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175222219 2023-04-22 14:53:46,532 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of cbd04a56b211199f6d2f4a000c3bb4f6/cf1 in testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:46,532 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-ae6318f1ccfb421eaebbac78986da928, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-3a4b982ef80e4b77a6f2a83e7ac1f59f, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-f4413a3d6e6f41328ec1e64be494fe81] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/.tmp, totalSize=34.9 K 2023-04-22 14:53:46,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-22 14:53:46,539 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; OpenRegionProcedure cbd04a56b211199f6d2f4a000c3bb4f6, server=jenkins-hbase4.apache.org,36267,1682175218583 in 540 msec 2023-04-22 14:53:46,544 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cbd04a56b211199f6d2f4a000c3bb4f6, ASSIGN in 706 msec 2023-04-22 14:53:46,545 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175222608 2023-04-22 14:53:46,546 INFO [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:46,545 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175222219 2023-04-22 14:53:46,553 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175222888 2023-04-22 14:53:46,553 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175222608 2023-04-22 14:53:46,554 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175222888 2023-04-22 14:53:46,554 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682175223129 2023-04-22 14:53:46,555 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-29414784ccf94b5396b1f647a1defac5, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175223373 2023-04-22 14:53:46,555 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682175223129 2023-04-22 14:53:46,556 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-7b6e534a381843d285e051b58fac6eed, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175223611 2023-04-22 14:53:46,556 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1 2023-04-22 14:53:46,556 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-ae6318f1ccfb421eaebbac78986da928, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175223373 2023-04-22 14:53:46,557 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-fc94602f35ab4569817df2912b29b06e, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175223868 2023-04-22 14:53:46,557 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1 2023-04-22 14:53:46,557 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-3a4b982ef80e4b77a6f2a83e7ac1f59f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175223611 2023-04-22 14:53:46,559 INFO [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ce3321b32c9468788bb2b99657f8761a columnFamilyName cf1 2023-04-22 14:53:46,559 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-f4413a3d6e6f41328ec1e64be494fe81, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175223868 2023-04-22 14:53:46,596 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50-bottom 2023-04-22 14:53:46,611 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] throttle.PressureAwareThroughputController(145): cbd04a56b211199f6d2f4a000c3bb4f6#cf2#compaction#14 average throughput is 0.26 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:46,614 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] throttle.PressureAwareThroughputController(145): cbd04a56b211199f6d2f4a000c3bb4f6#cf1#compaction#15 average throughput is 0.32 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:46,624 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5-bottom 2023-04-22 14:53:46,647 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f-bottom 2023-04-22 14:53:46,655 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/.tmp/cf2/1851cdea47af45f887a4ed8357bfa511 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/1851cdea47af45f887a4ed8357bfa511 2023-04-22 14:53:46,658 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f-bottom 2023-04-22 14:53:46,659 INFO [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(310): Store=ce3321b32c9468788bb2b99657f8761a/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:46,659 INFO [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:46,661 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2 2023-04-22 14:53:46,661 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2 2023-04-22 14:53:46,661 INFO [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ce3321b32c9468788bb2b99657f8761a columnFamilyName cf2 2023-04-22 14:53:46,683 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in cbd04a56b211199f6d2f4a000c3bb4f6/cf2 of cbd04a56b211199f6d2f4a000c3bb4f6 into 1851cdea47af45f887a4ed8357bfa511(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:46,685 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbd04a56b211199f6d2f4a000c3bb4f6: 2023-04-22 14:53:46,685 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6., storeName=cbd04a56b211199f6d2f4a000c3bb4f6/cf2, priority=9, startTime=1682175226501; duration=0sec 2023-04-22 14:53:46,686 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:46,687 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/.tmp/cf1/8872d1ed9ce2413b910e249f6f98cbbe as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/8872d1ed9ce2413b910e249f6f98cbbe 2023-04-22 14:53:46,692 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a-bottom 2023-04-22 14:53:46,703 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52-bottom 2023-04-22 14:53:46,707 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in cbd04a56b211199f6d2f4a000c3bb4f6/cf1 of cbd04a56b211199f6d2f4a000c3bb4f6 into 8872d1ed9ce2413b910e249f6f98cbbe(size=6.7 K), total size for store is 6.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:46,708 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for cbd04a56b211199f6d2f4a000c3bb4f6: 2023-04-22 14:53:46,708 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6., storeName=cbd04a56b211199f6d2f4a000c3bb4f6/cf1, priority=9, startTime=1682175226497; duration=0sec 2023-04-22 14:53:46,709 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:46,717 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849-bottom 2023-04-22 14:53:46,728 DEBUG [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610-bottom 2023-04-22 14:53:46,728 INFO [StoreOpener-ce3321b32c9468788bb2b99657f8761a-1] regionserver.HStore(310): Store=ce3321b32c9468788bb2b99657f8761a/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:46,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:46,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:46,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionUnevenDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:53:46,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:46,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ce3321b32c9468788bb2b99657f8761a; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64358964, jitterRate=-0.04097670316696167}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:53:46,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ce3321b32c9468788bb2b99657f8761a: 2023-04-22 14:53:46,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a., pid=18, masterSystemTime=1682175226147 2023-04-22 14:53:46,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:46,744 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:53:46,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:46,744 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:46,744 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1912): ce3321b32c9468788bb2b99657f8761a/cf1 is initiating minor compaction (all files) 2023-04-22 14:53:46,744 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-22 14:53:46,744 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ce3321b32c9468788bb2b99657f8761a/cf1 in testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:46,745 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/.tmp, totalSize=19.9 K 2023-04-22 14:53:46,745 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:46,745 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1912): ce3321b32c9468788bb2b99657f8761a/cf2 is initiating minor compaction (all files) 2023-04-22 14:53:46,745 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ce3321b32c9468788bb2b99657f8761a/cf2 in testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:46,746 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/.tmp, totalSize=19.9 K 2023-04-22 14:53:46,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:46,749 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:46,750 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175222219 2023-04-22 14:53:46,750 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=ce3321b32c9468788bb2b99657f8761a, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:46,756 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175222219 2023-04-22 14:53:46,756 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175226750"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175226750"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175226750"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175226750"}]},"ts":"1682175226750"} 2023-04-22 14:53:46,757 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175222608 2023-04-22 14:53:46,757 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175222608 2023-04-22 14:53:46,760 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175222888 2023-04-22 14:53:46,760 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175222888 2023-04-22 14:53:46,761 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175223129 2023-04-22 14:53:46,761 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175223129 2023-04-22 14:53:46,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=15 2023-04-22 14:53:46,767 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=15, state=SUCCESS; OpenRegionProcedure ce3321b32c9468788bb2b99657f8761a, server=jenkins-hbase4.apache.org,36267,1682175218583 in 757 msec 2023-04-22 14:53:46,773 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=12 2023-04-22 14:53:46,773 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ce3321b32c9468788bb2b99657f8761a, ASSIGN in 934 msec 2023-04-22 14:53:46,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 in 2.8700 sec 2023-04-22 14:53:46,782 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] throttle.PressureAwareThroughputController(145): ce3321b32c9468788bb2b99657f8761a#cf2#compaction#16 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:46,785 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] throttle.PressureAwareThroughputController(145): ce3321b32c9468788bb2b99657f8761a#cf1#compaction#17 average throughput is 0.43 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:46,842 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/.tmp/cf2/a5d2e862ce2b4efc9631347e6b17f99f as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/a5d2e862ce2b4efc9631347e6b17f99f 2023-04-22 14:53:46,857 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/.tmp/cf1/70c2ed27c7d3454a9719d91c0ca16f2e as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/70c2ed27c7d3454a9719d91c0ca16f2e 2023-04-22 14:53:46,861 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ce3321b32c9468788bb2b99657f8761a/cf2 of ce3321b32c9468788bb2b99657f8761a into a5d2e862ce2b4efc9631347e6b17f99f(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:46,862 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ce3321b32c9468788bb2b99657f8761a: 2023-04-22 14:53:46,862 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a., storeName=ce3321b32c9468788bb2b99657f8761a/cf2, priority=12, startTime=1682175226744; duration=0sec 2023-04-22 14:53:46,862 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:46,869 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ce3321b32c9468788bb2b99657f8761a/cf1 of ce3321b32c9468788bb2b99657f8761a into 70c2ed27c7d3454a9719d91c0ca16f2e(size=5.5 K), total size for store is 5.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:46,869 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ce3321b32c9468788bb2b99657f8761a: 2023-04-22 14:53:46,869 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a., storeName=ce3321b32c9468788bb2b99657f8761a/cf1, priority=12, startTime=1682175226743; duration=0sec 2023-04-22 14:53:46,869 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:46,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36267] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Scan size: 157 connection: 172.31.14.131:50214 deadline: 1682175286950, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099. is not online on jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:47,098 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-04-22 14:53:47,129 INFO [Listener at localhost/43403] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-04-22 14:53:47,134 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '26', inclusive 2023-04-22 14:53:47,195 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:53:47,214 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionUnevenDaughter 2023-04-22 14:53:47,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegionUnevenDaughter 2023-04-22 14:53:47,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:47,237 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175227237"}]},"ts":"1682175227237"} 2023-04-22 14:53:47,242 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLING in hbase:meta 2023-04-22 14:53:47,245 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionUnevenDaughter to state=DISABLING 2023-04-22 14:53:47,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-22 14:53:47,248 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ce3321b32c9468788bb2b99657f8761a, UNASSIGN}, {pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cbd04a56b211199f6d2f4a000c3bb4f6, UNASSIGN}] 2023-04-22 14:53:47,250 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cbd04a56b211199f6d2f4a000c3bb4f6, UNASSIGN 2023-04-22 14:53:47,252 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=cbd04a56b211199f6d2f4a000c3bb4f6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:47,253 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175227252"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175227252"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175227252"}]},"ts":"1682175227252"} 2023-04-22 14:53:47,256 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure cbd04a56b211199f6d2f4a000c3bb4f6, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:47,262 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=20, ppid=19, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ce3321b32c9468788bb2b99657f8761a, UNASSIGN 2023-04-22 14:53:47,264 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=ce3321b32c9468788bb2b99657f8761a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:47,264 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175227263"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175227263"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175227263"}]},"ts":"1682175227263"} 2023-04-22 14:53:47,267 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=20, state=RUNNABLE; CloseRegionProcedure ce3321b32c9468788bb2b99657f8761a, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:47,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-22 14:53:47,410 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:47,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ce3321b32c9468788bb2b99657f8761a, disabling compactions & flushes 2023-04-22 14:53:47,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:47,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:47,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. after waiting 0 ms 2023-04-22 14:53:47,411 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:47,421 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5-bottom] to archive 2023-04-22 14:53:47,425 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:53:47,433 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,436 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,436 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-22 14:53:47,438 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-22 14:53:47,439 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:53:47,439 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro Metrics about HBase RegionObservers 2023-04-22 14:53:47,439 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,441 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,482 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610-bottom] to archive 2023-04-22 14:53:47,483 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:53:47,487 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,490 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,495 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,498 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:53:47,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:53:47,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a. 2023-04-22 14:53:47,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ce3321b32c9468788bb2b99657f8761a: 2023-04-22 14:53:47,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:47,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:47,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing cbd04a56b211199f6d2f4a000c3bb4f6, disabling compactions & flushes 2023-04-22 14:53:47,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:47,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:47,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. after waiting 0 ms 2023-04-22 14:53:47,513 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:47,514 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=20 updating hbase:meta row=ce3321b32c9468788bb2b99657f8761a, regionState=CLOSED 2023-04-22 14:53:47,514 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175227514"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175227514"}]},"ts":"1682175227514"} 2023-04-22 14:53:47,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=20 2023-04-22 14:53:47,521 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=20, state=SUCCESS; CloseRegionProcedure ce3321b32c9468788bb2b99657f8761a, server=jenkins-hbase4.apache.org,36267,1682175218583 in 251 msec 2023-04-22 14:53:47,524 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=ce3321b32c9468788bb2b99657f8761a, UNASSIGN in 273 msec 2023-04-22 14:53:47,528 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-ae6318f1ccfb421eaebbac78986da928, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-3a4b982ef80e4b77a6f2a83e7ac1f59f, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-f4413a3d6e6f41328ec1e64be494fe81] to archive 2023-04-22 14:53:47,530 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:53:47,538 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/5a63098d55b14793a64159229531804f.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,541 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/88655a01a6794c30a88fb4b3f8d5dc2f.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,543 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/11b157fcfe7349bca3bc71853427cb50.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,545 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/1f5011b82cc645eba7885b8f9d4aceb5.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,547 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-ae6318f1ccfb421eaebbac78986da928 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-ae6318f1ccfb421eaebbac78986da928 2023-04-22 14:53:47,551 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-3a4b982ef80e4b77a6f2a83e7ac1f59f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-3a4b982ef80e4b77a6f2a83e7ac1f59f 2023-04-22 14:53:47,555 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-f4413a3d6e6f41328ec1e64be494fe81 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-f4413a3d6e6f41328ec1e64be494fe81 2023-04-22 14:53:47,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-22 14:53:47,562 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-29414784ccf94b5396b1f647a1defac5, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-7b6e534a381843d285e051b58fac6eed, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-fc94602f35ab4569817df2912b29b06e] to archive 2023-04-22 14:53:47,563 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:53:47,571 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/5d7f1432245b4bb19d5ef86844457849.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,573 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/558446f99181496dbf918ba699920f52.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,575 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/2178a13bef814adc82ef58ae3f82fc7a.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,576 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/cb259bd4795f45b18d809f71f95e1610.1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,578 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-29414784ccf94b5396b1f647a1defac5 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-29414784ccf94b5396b1f647a1defac5 2023-04-22 14:53:47,579 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-7b6e534a381843d285e051b58fac6eed to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-7b6e534a381843d285e051b58fac6eed 2023-04-22 14:53:47,581 DEBUG [StoreCloser-testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-fc94602f35ab4569817df2912b29b06e to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/testSplitTableRegionUnevenDaughter=1539dac144b8acabc3b8d8b035e2f099-fc94602f35ab4569817df2912b29b06e 2023-04-22 14:53:47,588 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:53:47,590 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:53:47,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6. 2023-04-22 14:53:47,592 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for cbd04a56b211199f6d2f4a000c3bb4f6: 2023-04-22 14:53:47,594 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:47,596 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=cbd04a56b211199f6d2f4a000c3bb4f6, regionState=CLOSED 2023-04-22 14:53:47,596 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1682175227596"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175227596"}]},"ts":"1682175227596"} 2023-04-22 14:53:47,604 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-22 14:53:47,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure cbd04a56b211199f6d2f4a000c3bb4f6, server=jenkins-hbase4.apache.org,36267,1682175218583 in 343 msec 2023-04-22 14:53:47,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=19 2023-04-22 14:53:47,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=19, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionUnevenDaughter, region=cbd04a56b211199f6d2f4a000c3bb4f6, UNASSIGN in 368 msec 2023-04-22 14:53:47,620 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175227620"}]},"ts":"1682175227620"} 2023-04-22 14:53:47,623 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionUnevenDaughter, state=DISABLED in hbase:meta 2023-04-22 14:53:47,624 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionUnevenDaughter to state=DISABLED 2023-04-22 14:53:47,632 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter in 404 msec 2023-04-22 14:53:47,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-22 14:53:47,862 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 19 completed 2023-04-22 14:53:47,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegionUnevenDaughter 2023-04-22 14:53:47,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:47,883 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=24, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:47,893 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=24, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:47,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 14:53:47,907 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:47,907 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:47,907 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:47,911 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/recovered.edits] 2023-04-22 14:53:47,911 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/recovered.edits] 2023-04-22 14:53:47,911 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/recovered.edits] 2023-04-22 14:53:47,925 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/11b157fcfe7349bca3bc71853427cb50 2023-04-22 14:53:47,927 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/1f5011b82cc645eba7885b8f9d4aceb5 2023-04-22 14:53:47,933 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/3a4b982ef80e4b77a6f2a83e7ac1f59f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/3a4b982ef80e4b77a6f2a83e7ac1f59f 2023-04-22 14:53:47,934 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/70c2ed27c7d3454a9719d91c0ca16f2e to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf1/70c2ed27c7d3454a9719d91c0ca16f2e 2023-04-22 14:53:47,934 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/8872d1ed9ce2413b910e249f6f98cbbe to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf1/8872d1ed9ce2413b910e249f6f98cbbe 2023-04-22 14:53:47,947 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/a5d2e862ce2b4efc9631347e6b17f99f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/cf2/a5d2e862ce2b4efc9631347e6b17f99f 2023-04-22 14:53:47,949 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/5a63098d55b14793a64159229531804f 2023-04-22 14:53:47,950 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/1851cdea47af45f887a4ed8357bfa511 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/cf2/1851cdea47af45f887a4ed8357bfa511 2023-04-22 14:53:47,951 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/88655a01a6794c30a88fb4b3f8d5dc2f 2023-04-22 14:53:47,953 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a/recovered.edits/90.seqid 2023-04-22 14:53:47,955 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/ce3321b32c9468788bb2b99657f8761a 2023-04-22 14:53:47,955 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/ae6318f1ccfb421eaebbac78986da928 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/ae6318f1ccfb421eaebbac78986da928 2023-04-22 14:53:47,962 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/f4413a3d6e6f41328ec1e64be494fe81 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf1/f4413a3d6e6f41328ec1e64be494fe81 2023-04-22 14:53:47,964 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6/recovered.edits/90.seqid 2023-04-22 14:53:47,966 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:47,970 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/2178a13bef814adc82ef58ae3f82fc7a 2023-04-22 14:53:47,977 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/29414784ccf94b5396b1f647a1defac5 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/29414784ccf94b5396b1f647a1defac5 2023-04-22 14:53:47,981 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/558446f99181496dbf918ba699920f52 2023-04-22 14:53:47,985 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/5d7f1432245b4bb19d5ef86844457849 2023-04-22 14:53:47,987 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/7b6e534a381843d285e051b58fac6eed to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/7b6e534a381843d285e051b58fac6eed 2023-04-22 14:53:47,994 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/cb259bd4795f45b18d809f71f95e1610 2023-04-22 14:53:47,996 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/fc94602f35ab4569817df2912b29b06e to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/cf2/fc94602f35ab4569817df2912b29b06e 2023-04-22 14:53:48,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 14:53:48,010 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/recovered.edits/85.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099/recovered.edits/85.seqid 2023-04-22 14:53:48,013 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionUnevenDaughter/1539dac144b8acabc3b8d8b035e2f099 2023-04-22 14:53:48,013 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionUnevenDaughter regions 2023-04-22 14:53:48,017 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=24, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:48,053 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionUnevenDaughter from hbase:meta 2023-04-22 14:53:48,125 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionUnevenDaughter' descriptor. 2023-04-22 14:53:48,127 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=24, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:48,127 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionUnevenDaughter' from region states. 2023-04-22 14:53:48,127 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175228127"}]},"ts":"9223372036854775807"} 2023-04-22 14:53:48,127 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175228127"}]},"ts":"9223372036854775807"} 2023-04-22 14:53:48,127 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175228127"}]},"ts":"9223372036854775807"} 2023-04-22 14:53:48,143 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-22 14:53:48,143 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 1539dac144b8acabc3b8d8b035e2f099, NAME => 'testSplitTableRegionUnevenDaughter,,1682175221707.1539dac144b8acabc3b8d8b035e2f099.', STARTKEY => '', ENDKEY => ''}, {ENCODED => ce3321b32c9468788bb2b99657f8761a, NAME => 'testSplitTableRegionUnevenDaughter,,1682175223902.ce3321b32c9468788bb2b99657f8761a.', STARTKEY => '', ENDKEY => '26'}, {ENCODED => cbd04a56b211199f6d2f4a000c3bb4f6, NAME => 'testSplitTableRegionUnevenDaughter,26,1682175223902.cbd04a56b211199f6d2f4a000c3bb4f6.', STARTKEY => '26', ENDKEY => ''}] 2023-04-22 14:53:48,143 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionUnevenDaughter' as deleted. 2023-04-22 14:53:48,144 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionUnevenDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175228143"}]},"ts":"9223372036854775807"} 2023-04-22 14:53:48,160 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionUnevenDaughter state from META 2023-04-22 14:53:48,163 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=24, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:48,165 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter in 294 msec 2023-04-22 14:53:48,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-22 14:53:48,203 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionUnevenDaughter, procId: 24 completed 2023-04-22 14:53:48,237 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionUnevenDaughter Thread=392 (was 360) Potentially hanging thread: hconnection-0x42387a87-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:51436 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:51386 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:51632 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:51616 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:50110 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:51654 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:58452 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:58532 [Waiting for operation #13] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:51642 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:54638 [Waiting for operation #19] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:49566 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:49612 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:49832 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:40048 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:1;jenkins-hbase4:36267-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:40066 [Waiting for operation #7] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:51612 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=820 (was 692) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=354 (was 316) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=12981 (was 13284) 2023-04-22 14:53:48,264 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=392, OpenFileDescriptor=820, MaxFileDescriptor=60000, SystemLoadAverage=354, ProcessCount=172, AvailableMemoryMB=12978 2023-04-22 14:53:48,264 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:53:48,282 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:53:48,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:53:48,285 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:48,289 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:53:48,290 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:53:48,293 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,294 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84 empty. 2023-04-22 14:53:48,294 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,295 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-04-22 14:53:48,342 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:48,344 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(7675): creating {ENCODED => b8acaab15ee577f218a114962304cb84, NAME => 'testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitWithoutPONR', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:53:48,392 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:48,392 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1604): Closing b8acaab15ee577f218a114962304cb84, disabling compactions & flushes 2023-04-22 14:53:48,392 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:48,392 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:48,393 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. after waiting 0 ms 2023-04-22 14:53:48,393 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:48,393 INFO [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:48,393 DEBUG [RegionOpenAndInit-testSplitWithoutPONR-pool-0] regionserver.HRegion(1558): Region close journal for b8acaab15ee577f218a114962304cb84: 2023-04-22 14:53:48,397 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:53:48,399 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175228399"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175228399"}]},"ts":"1682175228399"} 2023-04-22 14:53:48,402 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:53:48,404 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:53:48,404 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175228404"}]},"ts":"1682175228404"} 2023-04-22 14:53:48,411 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLING in hbase:meta 2023-04-22 14:53:48,415 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:53:48,418 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:53:48,419 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:53:48,419 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:53:48,419 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:53:48,419 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, ASSIGN}] 2023-04-22 14:53:48,423 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, ASSIGN 2023-04-22 14:53:48,424 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=26, ppid=25, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:53:48,575 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:53:48,576 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=b8acaab15ee577f218a114962304cb84, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:48,577 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175228576"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175228576"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175228576"}]},"ts":"1682175228576"} 2023-04-22 14:53:48,582 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=27, ppid=26, state=RUNNABLE; OpenRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:48,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:48,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b8acaab15ee577f218a114962304cb84, NAME => 'testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:48,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:53:48,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:48,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,745 INFO [StoreOpener-b8acaab15ee577f218a114962304cb84-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,747 DEBUG [StoreOpener-b8acaab15ee577f218a114962304cb84-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1 2023-04-22 14:53:48,747 DEBUG [StoreOpener-b8acaab15ee577f218a114962304cb84-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1 2023-04-22 14:53:48,747 INFO [StoreOpener-b8acaab15ee577f218a114962304cb84-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8acaab15ee577f218a114962304cb84 columnFamilyName cf1 2023-04-22 14:53:48,748 INFO [StoreOpener-b8acaab15ee577f218a114962304cb84-1] regionserver.HStore(310): Store=b8acaab15ee577f218a114962304cb84/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:48,748 INFO [StoreOpener-b8acaab15ee577f218a114962304cb84-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,751 DEBUG [StoreOpener-b8acaab15ee577f218a114962304cb84-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2 2023-04-22 14:53:48,751 DEBUG [StoreOpener-b8acaab15ee577f218a114962304cb84-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2 2023-04-22 14:53:48,757 INFO [StoreOpener-b8acaab15ee577f218a114962304cb84-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b8acaab15ee577f218a114962304cb84 columnFamilyName cf2 2023-04-22 14:53:48,758 INFO [StoreOpener-b8acaab15ee577f218a114962304cb84-1] regionserver.HStore(310): Store=b8acaab15ee577f218a114962304cb84/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:48,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,771 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:53:48,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:48,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:48,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b8acaab15ee577f218a114962304cb84; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65202899, jitterRate=-0.028401091694831848}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:53:48,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b8acaab15ee577f218a114962304cb84: 2023-04-22 14:53:48,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84., pid=27, masterSystemTime=1682175228736 2023-04-22 14:53:48,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:48,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:48,798 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=b8acaab15ee577f218a114962304cb84, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:48,798 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175228798"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175228798"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175228798"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175228798"}]},"ts":"1682175228798"} 2023-04-22 14:53:48,806 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=27, resume processing ppid=26 2023-04-22 14:53:48,807 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=26, state=SUCCESS; OpenRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 in 221 msec 2023-04-22 14:53:48,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-22 14:53:48,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, ASSIGN in 388 msec 2023-04-22 14:53:48,811 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:53:48,811 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175228811"}]},"ts":"1682175228811"} 2023-04-22 14:53:48,814 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=ENABLED in hbase:meta 2023-04-22 14:53:48,818 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=25, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitWithoutPONR execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:53:48,824 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR in 536 msec 2023-04-22 14:53:49,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:49,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-22 14:53:49,053 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-22 14:53:49,053 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,054 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-22 14:53:49,054 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-22 14:53:49,054 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,054 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,056 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,056 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,056 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,056 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,057 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,057 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,059 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,060 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,061 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,062 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,062 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,062 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,063 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:49,063 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:49,063 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:49,063 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. started... 2023-04-22 14:53:49,064 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b8acaab15ee577f218a114962304cb84 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-22 14:53:49,065 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,066 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,067 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,087 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/5d3038915c9d4e879e6f1c48bd070ad7 2023-04-22 14:53:49,115 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/2c9c93d70ecf49ac95c488011ed60e24 2023-04-22 14:53:49,125 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/5d3038915c9d4e879e6f1c48bd070ad7 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7 2023-04-22 14:53:49,132 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:53:49,133 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/2c9c93d70ecf49ac95c488011ed60e24 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24 2023-04-22 14:53:49,142 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:53:49,144 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for b8acaab15ee577f218a114962304cb84 in 80ms, sequenceid=6, compaction requested=false 2023-04-22 14:53:49,144 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitWithoutPONR' 2023-04-22 14:53:49,144 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b8acaab15ee577f218a114962304cb84: 2023-04-22 14:53:49,145 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:49,145 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:49,145 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:49,145 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,145 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,145 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,148 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,148 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,148 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,148 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-22 14:53:49,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@434a8a26[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:49,151 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-22 14:53:49,151 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,153 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,153 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:49,153 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:49,153 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:49,154 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,155 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:49,156 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,157 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:49,161 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,162 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:49,162 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,162 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:49,162 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-22 14:53:49,163 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:49,163 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-22 14:53:49,163 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:49,163 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:49,163 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-22 14:53:49,163 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:49,172 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,173 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,173 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,174 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,173 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,174 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,175 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,178 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,179 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,179 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,192 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-22 14:53:49,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:49,193 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,194 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,194 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,194 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,194 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,194 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,193 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,195 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:49,196 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:49,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,296 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:49,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:49,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:49,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:49,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-22 14:53:49,325 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-22 14:53:49,325 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,326 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-22 14:53:49,326 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-22 14:53:49,326 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,326 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,328 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,328 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,328 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,328 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,328 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,328 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:49,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:49,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,330 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,331 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:49,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:49,337 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,339 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,339 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,338 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,339 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:49,340 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:49,340 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:49,340 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. started... 2023-04-22 14:53:49,340 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b8acaab15ee577f218a114962304cb84 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:49,342 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,343 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,383 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/0d39e311ab844856b36212e9324b5524 2023-04-22 14:53:49,404 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/1ccccd66690041609009da38872649da 2023-04-22 14:53:49,412 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/0d39e311ab844856b36212e9324b5524 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524 2023-04-22 14:53:49,419 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:53:49,421 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/1ccccd66690041609009da38872649da as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da 2023-04-22 14:53:49,432 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:53:49,433 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for b8acaab15ee577f218a114962304cb84 in 93ms, sequenceid=19, compaction requested=false 2023-04-22 14:53:49,433 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b8acaab15ee577f218a114962304cb84: 2023-04-22 14:53:49,433 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,436 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,436 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,437 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,437 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-22 14:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38d19430[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:49,439 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-22 14:53:49,439 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,441 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,441 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,441 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:49,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:49,441 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:49,441 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,441 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:49,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:49,453 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,454 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:49,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-22 14:53:49,463 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-22 14:53:49,463 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:49,463 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:49,463 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-22 14:53:49,463 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:49,466 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,466 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,466 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,466 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,466 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,472 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,472 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,480 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,482 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,483 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,486 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,487 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,487 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,492 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,492 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,492 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,492 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,492 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,492 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-22 14:53:49,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:49,493 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,493 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,492 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,493 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,493 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,494 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,494 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,493 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:49,494 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,494 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,494 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,494 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,494 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,494 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:49,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,594 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:49,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:49,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:49,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:49,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-22 14:53:49,632 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-22 14:53:49,632 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,633 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-22 14:53:49,633 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-22 14:53:49,634 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,634 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,635 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,635 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,636 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,636 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,636 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,636 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:49,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:49,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:49,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,644 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,644 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:49,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,652 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,654 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,655 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:49,656 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:49,656 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:49,656 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. started... 2023-04-22 14:53:49,656 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b8acaab15ee577f218a114962304cb84 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:49,657 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,660 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,664 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,665 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,665 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,686 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/b95bdf6639764f25b10d00c627ed4a7a 2023-04-22 14:53:49,738 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/afd8aad850e34c97bdae852fd2a0c6be 2023-04-22 14:53:49,746 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/b95bdf6639764f25b10d00c627ed4a7a as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a 2023-04-22 14:53:49,754 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:53:49,755 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/afd8aad850e34c97bdae852fd2a0c6be as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be 2023-04-22 14:53:49,761 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:53:49,762 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for b8acaab15ee577f218a114962304cb84 in 106ms, sequenceid=32, compaction requested=true 2023-04-22 14:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b8acaab15ee577f218a114962304cb84: 2023-04-22 14:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,765 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-22 14:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7788fc4b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:49,768 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-22 14:53:49,768 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,769 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,769 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,769 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:49,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:49,770 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,770 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:49,773 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:49,773 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,773 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:49,774 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:49,774 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:49,774 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,774 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,774 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,774 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:49,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-22 14:53:49,780 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-22 14:53:49,780 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:49,780 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:49,780 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-22 14:53:49,780 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:49,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,782 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:49,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,783 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,783 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,784 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,785 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:49,788 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:49,788 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,789 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,792 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,792 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,792 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,792 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,792 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,793 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,793 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,792 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,793 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,793 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,793 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:49,793 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,793 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:49,793 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,794 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:49,794 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:49,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:49,808 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,808 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:49,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-22 14:53:49,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:49,809 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:49,810 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:49,910 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:49,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:49,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:49,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:49,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-22 14:53:49,952 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-22 14:53:49,952 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,953 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-22 14:53:49,953 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-22 14:53:49,954 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,954 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,964 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,964 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,964 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,964 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:49,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,965 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:49,965 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:49,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:49,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:49,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,968 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,968 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,969 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:49,969 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:49,969 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:49,974 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:49,975 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:49,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:49,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,976 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:49,993 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:50,010 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:50,011 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:50,011 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. started... 2023-04-22 14:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:50,011 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:50,011 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b8acaab15ee577f218a114962304cb84 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,013 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,014 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,067 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8 2023-04-22 14:53:50,116 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/d4940bb491214ece9cbdb8f596d7a186 2023-04-22 14:53:50,123 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8 2023-04-22 14:53:50,140 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:53:50,143 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/d4940bb491214ece9cbdb8f596d7a186 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186 2023-04-22 14:53:50,152 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:53:50,154 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for b8acaab15ee577f218a114962304cb84 in 143ms, sequenceid=45, compaction requested=true 2023-04-22 14:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b8acaab15ee577f218a114962304cb84: 2023-04-22 14:53:50,154 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:50,155 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:50,158 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-22 14:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77ad8264[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:50,163 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-22 14:53:50,163 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:50,165 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,165 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:50,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:50,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:50,169 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:50,172 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:50,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-22 14:53:50,180 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-22 14:53:50,180 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:50,181 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:50,181 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-22 14:53:50,181 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:50,184 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,184 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,184 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,184 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,184 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,185 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,186 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,184 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,184 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,186 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,192 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,193 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,193 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-22 14:53:50,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:50,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:50,203 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,202 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,304 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:50,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:50,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:50,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:50,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-22 14:53:50,355 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-22 14:53:50,355 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,355 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-22 14:53:50,355 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-22 14:53:50,356 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,356 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,357 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,357 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,357 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,357 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,362 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,362 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:50,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:50,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:50,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:50,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:50,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:50,364 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:50,364 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,364 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:50,364 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:50,365 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:50,365 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:50,365 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:50,365 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. started... 2023-04-22 14:53:50,365 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:50,365 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:50,365 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,365 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:50,365 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:50,365 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b8acaab15ee577f218a114962304cb84 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:50,365 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,367 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:50,367 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:50,369 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,388 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/ff7154718348403b81f4561a26fc586e 2023-04-22 14:53:50,467 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/ec34b2adc15d4790a2210b4a2ad3a096 2023-04-22 14:53:50,476 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/ff7154718348403b81f4561a26fc586e as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e 2023-04-22 14:53:50,486 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:53:50,488 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/ec34b2adc15d4790a2210b4a2ad3a096 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096 2023-04-22 14:53:50,505 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:53:50,507 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for b8acaab15ee577f218a114962304cb84 in 142ms, sequenceid=58, compaction requested=true 2023-04-22 14:53:50,507 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b8acaab15ee577f218a114962304cb84: 2023-04-22 14:53:50,507 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:50,507 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:50,507 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:50,507 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,507 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:50,507 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:50,509 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,510 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,511 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,511 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-22 14:53:50,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60213d0d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:50,513 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-22 14:53:50,513 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,514 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,514 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,514 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,515 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,515 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:50,515 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:50,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,515 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:50,515 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,515 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:50,515 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:50,515 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:50,515 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:50,515 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:50,515 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:50,517 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,517 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,517 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,517 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:50,523 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:50,523 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,523 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:50,524 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:50,524 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,524 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-22 14:53:50,530 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-22 14:53:50,530 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:50,530 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:50,530 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-22 14:53:50,530 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:50,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,534 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,534 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,535 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,538 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,539 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,539 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,549 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,549 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,549 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,549 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,549 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,550 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,549 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-22 14:53:50,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:50,549 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:50,549 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,551 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:50,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,550 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,651 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:50,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:50,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitWithoutPONR was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:50,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitWithoutPONR 2023-04-22 14:53:50,684 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitWithoutPONR' 2023-04-22 14:53:50,684 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,684 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitWithoutPONR' starting 'acquire' 2023-04-22 14:53:50,684 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitWithoutPONR', kicking off acquire phase on members. 2023-04-22 14:53:50,685 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,685 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,686 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,686 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,686 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:50,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:50,688 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitWithoutPONR 2023-04-22 14:53:50,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:50,694 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,695 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,695 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:50,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitWithoutPONR is still present, removing 2023-04-22 14:53:50,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:50,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitWithoutPONR 2023-04-22 14:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitWithoutPONR' with timeout 60000ms 2023-04-22 14:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:50,705 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:50,704 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:50,705 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitWithoutPONR' starting 'acquire' stage 2023-04-22 14:53:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,708 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,716 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:50,716 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:50,718 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:50,718 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. started... 2023-04-22 14:53:50,718 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b8acaab15ee577f218a114962304cb84 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:50,763 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/3d63774c5ca5488eb4a7706eb2f81de8 2023-04-22 14:53:50,788 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/c013e8636d7b46fe951dbbfe0bf01f23 2023-04-22 14:53:50,804 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/3d63774c5ca5488eb4a7706eb2f81de8 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8 2023-04-22 14:53:50,812 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:53:50,814 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/c013e8636d7b46fe951dbbfe0bf01f23 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23 2023-04-22 14:53:50,825 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:53:50,827 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for b8acaab15ee577f218a114962304cb84 in 109ms, sequenceid=71, compaction requested=true 2023-04-22 14:53:50,827 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b8acaab15ee577f218a114962304cb84: 2023-04-22 14:53:50,827 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:50,827 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:50,827 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:50,827 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,827 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitWithoutPONR' locally acquired 2023-04-22 14:53:50,827 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitWithoutPONR) in zk 2023-04-22 14:53:50,835 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,835 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,836 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,836 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitWithoutPONR' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitWithoutPONR' on coordinator 2023-04-22 14:53:50,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5465552d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:50,839 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitWithoutPONR' starting 'in-barrier' execution. 2023-04-22 14:53:50,839 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,841 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,841 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,841 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:50,841 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,842 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitWithoutPONR' received 'reached' from coordinator. 2023-04-22 14:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitWithoutPONR' locally completed 2023-04-22 14:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:50,842 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitWithoutPONR' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:50,844 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,845 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:50,845 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,845 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:50,847 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitWithoutPONR' has notified controller of completion 2023-04-22 14:53:50,847 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,847 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitWithoutPONR' completed. 2023-04-22 14:53:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitWithoutPONR' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitWithoutPONR', counting down latch. Waiting for 0 more 2023-04-22 14:53:50,851 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitWithoutPONR' execution completed 2023-04-22 14:53:50,851 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:50,851 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:50,851 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitWithoutPONR 2023-04-22 14:53:50,851 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitWithoutPONRincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:50,853 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,853 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,853 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,853 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,853 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:50,853 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,854 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:50,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,855 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,856 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:50,856 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,858 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:50,859 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitWithoutPONR 2023-04-22 14:53:50,859 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,863 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,863 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,863 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,863 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:50,863 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitWithoutPONR' 2023-04-22 14:53:50,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:50,863 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitWithoutPONR 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitWithoutPONR 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitWithoutPONR 2023-04-22 14:53:50,863 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:50,863 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:50,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,864 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitWithoutPONR'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:50,864 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:50,865 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:50,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:50,965 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:50,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:51,238 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-04-22 14:53:51,241 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:51,242 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,243 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,244 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,244 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,245 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:51,246 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,246 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:51,246 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:51,493 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,494 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:51,494 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:51,494 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:51,496 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] 2023-04-22 14:53:51,497 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:51,497 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:51,497 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:51,498 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:51,498 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:53:51,506 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-04-22 14:53:51,508 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:51,509 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:51,509 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:51,510 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:51,510 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:51,511 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:51,511 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:51,512 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:51,513 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:51,513 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,513 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:51,513 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,514 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,514 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,514 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 15 msec 2023-04-22 14:53:51,516 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:51,516 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:51,526 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175261526 2023-04-22 14:53:51,528 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,528 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:51,532 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:51,533 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:51,533 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:51,533 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175291533 2023-04-22 14:53:51,534 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175351534 2023-04-22 14:53:51,534 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-04-22 14:53:51,534 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:53:51,537 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:51,537 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,537 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:51,537 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:51,659 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:53:51,784 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,784 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:51,784 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:51,785 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:51,786 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:53:51,786 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:51,786 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:51,786 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:51,786 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:51,787 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:53:51,789 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-04-22 14:53:51,790 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:51,791 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:51,791 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:51,791 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:51,792 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:51,793 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:51,793 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:51,793 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:51,794 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:51,794 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,795 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:51,795 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,795 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,795 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,795 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-22 14:53:51,796 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:51,797 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:51,805 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175261805 2023-04-22 14:53:51,805 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:51,808 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,810 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:51,811 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:51,811 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:51,811 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175291811 2023-04-22 14:53:51,811 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175351811 2023-04-22 14:53:51,811 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-04-22 14:53:51,813 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:53:51,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN}] 2023-04-22 14:53:51,816 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:51,816 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:51,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:51,816 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:52,062 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,062 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:52,062 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:52,062 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:52,063 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:53:52,064 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:52,064 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:52,064 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:52,064 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:52,065 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:53:52,066 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-04-22 14:53:52,066 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:52,067 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:52,067 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:52,067 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:52,069 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:52,069 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:52,069 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:52,069 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:52,070 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:52,070 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,070 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:52,070 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,071 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,071 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,071 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-22 14:53:52,072 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:52,073 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:52,073 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175262073 2023-04-22 14:53:52,075 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,075 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:52,078 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:52,079 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:52,079 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:52,079 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175292079 2023-04-22 14:53:52,079 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175352079 2023-04-22 14:53:52,079 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-22 14:53:52,080 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN}] 2023-04-22 14:53:52,080 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:53:52,082 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:52,083 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=b8acaab15ee577f218a114962304cb84, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:52,084 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175232083"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175232083"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175232083"}]},"ts":"1682175232083"} 2023-04-22 14:53:52,089 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:52,089 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:52,089 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:52,089 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:52,093 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:52,329 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,330 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:52,330 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:52,330 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:52,331 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:53:52,332 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:52,332 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:52,332 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:52,332 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:52,333 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:53:52,334 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:53:52,335 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-22 14:53:52,335 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:52,336 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:52,336 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:52,336 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:52,337 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:52,337 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:52,337 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:52,337 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:52,338 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:52,339 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,339 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:52,339 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:52,339 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,339 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,339 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,339 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:52,340 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:52,340 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-22 14:53:52,343 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:52,344 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:52,344 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84 to restore RIT 2023-04-22 14:53:52,346 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175262346 2023-04-22 14:53:52,348 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:52,348 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:52,358 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:52,358 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=CLOSING, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:52,358 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:52,359 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175292358 2023-04-22 14:53:52,359 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175352359 2023-04-22 14:53:52,359 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 10 msec 2023-04-22 14:53:52,360 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=b8acaab15ee577f218a114962304cb84, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:52,360 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175232360"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175232360"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175232360"}]},"ts":"1682175232360"} 2023-04-22 14:53:52,363 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:52,363 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:53:52,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:52,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b8acaab15ee577f218a114962304cb84, disabling compactions & flushes 2023-04-22 14:53:52,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:52,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:52,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. after waiting 0 ms 2023-04-22 14:53:52,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:52,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b8acaab15ee577f218a114962304cb84 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-22 14:53:52,539 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/3c4da0b13cc04b9ba7dbedf28988cf42 2023-04-22 14:53:52,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/fd5e75f6a08441c08e84e47e72cfc20f 2023-04-22 14:53:52,576 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf1/3c4da0b13cc04b9ba7dbedf28988cf42 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42 2023-04-22 14:53:52,582 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:53:52,583 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/.tmp/cf2/fd5e75f6a08441c08e84e47e72cfc20f as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f 2023-04-22 14:53:52,590 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:53:52,592 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for b8acaab15ee577f218a114962304cb84 in 73ms, sequenceid=82, compaction requested=true 2023-04-22 14:53:52,615 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-22 14:53:52,616 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:53:52,617 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. 2023-04-22 14:53:52,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b8acaab15ee577f218a114962304cb84: 2023-04-22 14:53:52,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:52,622 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=b8acaab15ee577f218a114962304cb84, regionState=CLOSED 2023-04-22 14:53:52,622 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175232622"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175232622"}]},"ts":"1682175232622"} 2023-04-22 14:53:52,624 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:52,624 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:52,624 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:52,624 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=30, ppid=29, state=SUCCESS, locked=true; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:52,859 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,859 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:52,859 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:52,860 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:52,860 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:53:52,860 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:52,860 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:52,861 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:52,861 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:52,862 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:53:52,863 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:53:52,864 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-04-22 14:53:52,864 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-04-22 14:53:52,864 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:52,865 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:52,865 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:52,865 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:52,866 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:52,866 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:52,866 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:52,867 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:52,867 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:52,868 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,868 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:52,868 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:52,869 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:52,869 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,869 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,869 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:52,869 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:52,869 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:52,869 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:52,869 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-22 14:53:52,870 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:52,871 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:52,871 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84 to restore RIT 2023-04-22 14:53:52,888 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175262888 2023-04-22 14:53:52,893 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=30, ppid=29, state=RUNNABLE; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:52,893 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:52,896 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:52,897 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:52,897 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:52,897 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175292897 2023-04-22 14:53:52,897 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175352897 2023-04-22 14:53:52,897 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-22 14:53:52,898 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=b8acaab15ee577f218a114962304cb84, regionState=CLOSED, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:52,898 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175232898"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175232898"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175232898"}]},"ts":"1682175232898"} 2023-04-22 14:53:52,902 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:53:52,906 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-22 14:53:52,906 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 in 539 msec 2023-04-22 14:53:52,907 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:52,907 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:52,907 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:52,907 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=29, ppid=28, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:53,148 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,148 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:53,148 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:53,148 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:53,149 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:53:53,150 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:53,150 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:53,150 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:53,150 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:53,151 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:53:53,152 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:53:53,153 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-04-22 14:53:53,153 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:53:53,153 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:53,154 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:53,154 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:53,154 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:53,155 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:53,155 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:53,155 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:53,155 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:53,156 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:53,156 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,157 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:53,157 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:53,157 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:53,157 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,157 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,157 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,157 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:53,157 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:53,157 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-22 14:53:53,158 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:53,158 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:53,159 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN to state=OFFLINE, location=null, table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84 to restore RIT 2023-04-22 14:53:53,160 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175263160 2023-04-22 14:53:53,161 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:53,161 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:53,165 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:53,165 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:53,165 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:53,165 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175293165 2023-04-22 14:53:53,165 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175353165 2023-04-22 14:53:53,166 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-22 14:53:53,168 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:53,170 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:53:53,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-22 14:53:53,177 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN in 1.0910 sec 2023-04-22 14:53:53,180 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:53,181 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,181 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:53,181 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:53,416 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,416 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:53,416 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:53,416 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:53,416 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:53:53,417 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:53,417 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:53,417 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:53,417 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:53,418 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:53:53,419 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:53:53,420 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:53:53,420 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:53:53,420 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:53,421 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:53,421 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:53,421 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:53,422 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:53,422 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:53,423 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:53,423 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:53,424 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:53,425 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,425 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:53,425 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:53,425 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:53,425 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:53,425 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:53,426 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-22 14:53:53,427 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:53,428 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:53,428 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175263428 2023-04-22 14:53:53,430 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:53,430 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,433 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:53,433 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:53,433 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:53,433 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175293433 2023-04-22 14:53:53,433 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175353433 2023-04-22 14:53:53,434 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:53:53,434 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,434 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,434 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:53,443 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:53:53,457 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=b8acaab15ee577f218a114962304cb84, threads=14 2023-04-22 14:53:53,458 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,458 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,459 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,459 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,459 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,459 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,459 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,464 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,468 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,471 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,472 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,472 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,476 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,476 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,778 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,778 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,787 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,823 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,868 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,868 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,904 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,956 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,970 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,973 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,975 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:53,978 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,376 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,383 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,383 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region b8acaab15ee577f218a114962304cb84 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-22 14:53:54,497 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:54,497 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:54,497 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:54,498 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:54,684 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:54,684 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:54,684 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:54,685 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:54,686 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:53:54,686 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:54,686 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:54,686 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:54,687 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:54,688 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-22 14:53:54,689 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-04-22 14:53:54,690 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:53:54,690 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:53:54,690 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:54,691 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:54,691 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:54,692 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:54,693 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:54,693 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:54,694 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:54,694 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:54,695 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:54,696 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:54,696 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:54,696 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:54,696 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:54,697 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:54,697 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:54,697 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:54,697 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:54,697 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:54,697 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 9 msec 2023-04-22 14:53:54,700 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:54,701 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:54,702 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175264702 2023-04-22 14:53:54,703 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:54,703 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:54,706 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:54,706 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:54,707 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:54,707 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175294707 2023-04-22 14:53:54,707 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175354707 2023-04-22 14:53:54,707 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:53:54,721 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=28 splitting 14 storefiles, region=b8acaab15ee577f218a114962304cb84, threads=14 2023-04-22 14:53:54,723 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,723 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,724 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,728 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,731 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,732 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,749 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,754 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,759 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,762 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,765 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,771 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,771 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:54,776 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=28 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,132 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,176 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,176 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,200 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,201 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,279 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,280 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,298 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,308 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,322 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,330 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,332 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,342 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,344 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=28 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186 for region: b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:55,344 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=28 split storefiles for region b8acaab15ee577f218a114962304cb84 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-22 14:53:55,496 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:53:55,510 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:53:55,528 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:53:55,528 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:55,529 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,529 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:55,529 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:55,707 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,708 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:55,708 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:55,708 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:55,709 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:53:55,709 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:55,709 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:55,709 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:55,710 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:55,710 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:53:55,712 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-04-22 14:53:55,712 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:53:55,712 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:53:55,712 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:55,713 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:55,713 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:55,713 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:55,713 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:55,714 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:55,714 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:55,714 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:55,716 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:55,716 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,717 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:55,717 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:55,717 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:55,717 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,717 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,717 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,717 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:55,717 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:55,718 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-22 14:53:55,718 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:55,719 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:55,720 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175265720 2023-04-22 14:53:55,721 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,721 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:55,727 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:55,727 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:55,727 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:55,727 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175295727 2023-04-22 14:53:55,727 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175355727 2023-04-22 14:53:55,727 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-04-22 14:53:55,731 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:53:55,735 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:53:55,735 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,735 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:53:55,735 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:53:55,978 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:53:55,978 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:53:55,978 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:53:55,978 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:53:55,979 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:53:55,980 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:53:55,980 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:53:55,980 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:53:55,980 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:53:55,981 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:53:55,982 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=28, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-04-22 14:53:55,982 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=29, ppid=28, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:53:55,982 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=30, ppid=29, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:53:55,983 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:53:55,983 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:53:55,983 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:53:55,984 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:53:55,984 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:55,985 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:53:55,985 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:55,986 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:53:55,987 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:55,988 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,988 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN 2023-04-22 14:53:55,988 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:55,988 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:53:55,988 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,989 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,989 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:55,989 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=b8acaab15ee577f218a114962304cb84, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:55,989 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=30, ppid=29, state=SUCCESS; CloseRegionProcedure b8acaab15ee577f218a114962304cb84, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:53:55,989 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-22 14:53:55,990 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:53:55,991 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:53:55,995 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175265995 2023-04-22 14:53:55,995 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:53:55,999 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=28, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:56,001 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:56,001 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=b8acaab15ee577f218a114962304cb84, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:53:56,001 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:53:56,001 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175296001 2023-04-22 14:53:56,002 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175356002 2023-04-22 14:53:56,002 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-04-22 14:53:56,008 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175236008"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1682175236008"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1682175236008"}]},"ts":"1682175236008"} 2023-04-22 14:53:56,009 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175236008"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175236008"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175236008"}]},"ts":"1682175236008"} 2023-04-22 14:53:56,009 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175236008"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175236008"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175236008"}]},"ts":"1682175236008"} 2023-04-22 14:53:56,026 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a8630043203581c85952b1c770f58cd9, ASSIGN}, {pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=bcf49cb18a7738ad1232f3887a8bd9af, ASSIGN}] 2023-04-22 14:53:56,028 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=bcf49cb18a7738ad1232f3887a8bd9af, ASSIGN 2023-04-22 14:53:56,029 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=32, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=bcf49cb18a7738ad1232f3887a8bd9af, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:53:56,030 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a8630043203581c85952b1c770f58cd9, ASSIGN 2023-04-22 14:53:56,031 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=31, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a8630043203581c85952b1c770f58cd9, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:53:56,179 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-22 14:53:56,181 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=a8630043203581c85952b1c770f58cd9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:56,182 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175236181"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175236181"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175236181"}]},"ts":"1682175236181"} 2023-04-22 14:53:56,184 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=31, state=RUNNABLE; OpenRegionProcedure a8630043203581c85952b1c770f58cd9, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:56,188 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=bcf49cb18a7738ad1232f3887a8bd9af, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:56,188 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175236188"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175236188"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175236188"}]},"ts":"1682175236188"} 2023-04-22 14:53:56,190 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=32, state=RUNNABLE; OpenRegionProcedure bcf49cb18a7738ad1232f3887a8bd9af, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:56,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:56,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a8630043203581c85952b1c770f58cd9, NAME => 'testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.', STARTKEY => '', ENDKEY => '41'} 2023-04-22 14:53:56,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:53:56,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:56,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:56,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:56,342 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:56,344 INFO [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:56,345 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1 2023-04-22 14:53:56,345 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1 2023-04-22 14:53:56,345 INFO [StoreOpener-a8630043203581c85952b1c770f58cd9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8630043203581c85952b1c770f58cd9 columnFamilyName cf1 2023-04-22 14:53:56,372 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524-bottom 2023-04-22 14:53:56,387 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42-bottom 2023-04-22 14:53:56,389 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8-bottom 2023-04-22 14:53:56,399 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7-bottom 2023-04-22 14:53:56,406 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8-bottom 2023-04-22 14:53:56,413 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a-bottom 2023-04-22 14:53:56,420 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e-bottom 2023-04-22 14:53:56,420 INFO [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(310): Store=a8630043203581c85952b1c770f58cd9/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:56,420 INFO [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:56,421 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2 2023-04-22 14:53:56,422 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2 2023-04-22 14:53:56,422 INFO [StoreOpener-a8630043203581c85952b1c770f58cd9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a8630043203581c85952b1c770f58cd9 columnFamilyName cf2 2023-04-22 14:53:56,443 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da-bottom 2023-04-22 14:53:56,450 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24-bottom 2023-04-22 14:53:56,457 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be-bottom 2023-04-22 14:53:56,463 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23-bottom 2023-04-22 14:53:56,476 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186-bottom 2023-04-22 14:53:56,483 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096-bottom 2023-04-22 14:53:56,494 DEBUG [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f-bottom 2023-04-22 14:53:56,494 INFO [StoreOpener-a8630043203581c85952b1c770f58cd9-1] regionserver.HStore(310): Store=a8630043203581c85952b1c770f58cd9/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:56,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:56,497 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:56,498 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:53:56,500 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:56,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a8630043203581c85952b1c770f58cd9; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67966315, jitterRate=0.012777015566825867}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:53:56,501 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a8630043203581c85952b1c770f58cd9: 2023-04-22 14:53:56,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9., pid=33, masterSystemTime=1682175236337 2023-04-22 14:53:56,503 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:56,504 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:53:56,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:56,504 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:56,504 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1912): a8630043203581c85952b1c770f58cd9/cf1 is initiating minor compaction (all files) 2023-04-22 14:53:56,504 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:53:56,504 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a8630043203581c85952b1c770f58cd9/cf1 in testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:56,504 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:56,504 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1912): a8630043203581c85952b1c770f58cd9/cf2 is initiating minor compaction (all files) 2023-04-22 14:53:56,504 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a8630043203581c85952b1c770f58cd9/cf2 in testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:56,505 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/.tmp, totalSize=34.9 K 2023-04-22 14:53:56,505 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/.tmp, totalSize=34.9 K 2023-04-22 14:53:56,506 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175229040 2023-04-22 14:53:56,506 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175229040 2023-04-22 14:53:56,506 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175229298 2023-04-22 14:53:56,507 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175229298 2023-04-22 14:53:56,507 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175229598 2023-04-22 14:53:56,508 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175229598 2023-04-22 14:53:56,508 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175229912 2023-04-22 14:53:56,508 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175229912 2023-04-22 14:53:56,509 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175230309 2023-04-22 14:53:56,509 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175230309 2023-04-22 14:53:56,510 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175230655 2023-04-22 14:53:56,510 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175230655 2023-04-22 14:53:56,510 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175230968 2023-04-22 14:53:56,511 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175230968 2023-04-22 14:53:56,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:56,511 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:56,511 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:56,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => bcf49cb18a7738ad1232f3887a8bd9af, NAME => 'testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.', STARTKEY => '41', ENDKEY => ''} 2023-04-22 14:53:56,512 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:53:56,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitWithoutPONR bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:56,512 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=31 updating hbase:meta row=a8630043203581c85952b1c770f58cd9, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:56,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:56,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:56,512 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:56,512 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175236512"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175236512"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175236512"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175236512"}]},"ts":"1682175236512"} 2023-04-22 14:53:56,514 INFO [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:56,516 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1 2023-04-22 14:53:56,517 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1 2023-04-22 14:53:56,518 INFO [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bcf49cb18a7738ad1232f3887a8bd9af columnFamilyName cf1 2023-04-22 14:53:56,518 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=31 2023-04-22 14:53:56,518 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=31, state=SUCCESS; OpenRegionProcedure a8630043203581c85952b1c770f58cd9, server=jenkins-hbase4.apache.org,36267,1682175218583 in 331 msec 2023-04-22 14:53:56,520 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a8630043203581c85952b1c770f58cd9, ASSIGN in 495 msec 2023-04-22 14:53:56,533 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] throttle.PressureAwareThroughputController(145): a8630043203581c85952b1c770f58cd9#cf2#compaction#32 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:56,550 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524-top 2023-04-22 14:53:56,558 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42-top 2023-04-22 14:53:56,575 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] throttle.PressureAwareThroughputController(145): a8630043203581c85952b1c770f58cd9#cf1#compaction#33 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:56,595 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8-top 2023-04-22 14:53:56,601 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7-top 2023-04-22 14:53:56,607 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8-top 2023-04-22 14:53:56,633 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a-top 2023-04-22 14:53:56,640 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/.tmp/cf2/10a4504351a34451a127ba8395850699 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/10a4504351a34451a127ba8395850699 2023-04-22 14:53:56,646 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e-top 2023-04-22 14:53:56,656 INFO [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(310): Store=bcf49cb18a7738ad1232f3887a8bd9af/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:56,656 INFO [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:56,658 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2 2023-04-22 14:53:56,658 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2 2023-04-22 14:53:56,658 INFO [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region bcf49cb18a7738ad1232f3887a8bd9af columnFamilyName cf2 2023-04-22 14:53:56,670 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a8630043203581c85952b1c770f58cd9/cf2 of a8630043203581c85952b1c770f58cd9 into 10a4504351a34451a127ba8395850699(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:56,670 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8630043203581c85952b1c770f58cd9: 2023-04-22 14:53:56,670 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9., storeName=a8630043203581c85952b1c770f58cd9/cf2, priority=9, startTime=1682175236504; duration=0sec 2023-04-22 14:53:56,670 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:56,677 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/.tmp/cf1/a5a6de66b1324674aa4350248fc88ee1 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/a5a6de66b1324674aa4350248fc88ee1 2023-04-22 14:53:56,688 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in a8630043203581c85952b1c770f58cd9/cf1 of a8630043203581c85952b1c770f58cd9 into a5a6de66b1324674aa4350248fc88ee1(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:56,688 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a8630043203581c85952b1c770f58cd9: 2023-04-22 14:53:56,688 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9., storeName=a8630043203581c85952b1c770f58cd9/cf1, priority=9, startTime=1682175236503; duration=0sec 2023-04-22 14:53:56,688 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:56,690 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da-top 2023-04-22 14:53:56,698 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24-top 2023-04-22 14:53:56,702 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be-top 2023-04-22 14:53:56,709 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23-top 2023-04-22 14:53:56,716 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186-top 2023-04-22 14:53:56,722 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096-top 2023-04-22 14:53:56,734 DEBUG [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f-top 2023-04-22 14:53:56,734 INFO [StoreOpener-bcf49cb18a7738ad1232f3887a8bd9af-1] regionserver.HStore(310): Store=bcf49cb18a7738ad1232f3887a8bd9af/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:56,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:56,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:56,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitWithoutPONR descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:53:56,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:56,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened bcf49cb18a7738ad1232f3887a8bd9af; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68248203, jitterRate=0.016977474093437195}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:53:56,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for bcf49cb18a7738ad1232f3887a8bd9af: 2023-04-22 14:53:56,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af., pid=34, masterSystemTime=1682175236337 2023-04-22 14:53:56,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:56,744 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:53:56,744 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:53:56,744 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:56,744 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:56,745 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1912): bcf49cb18a7738ad1232f3887a8bd9af/cf2 is initiating minor compaction (all files) 2023-04-22 14:53:56,745 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2259): Starting compaction of bcf49cb18a7738ad1232f3887a8bd9af/cf2 in testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:56,745 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f-top] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/.tmp, totalSize=34.9 K 2023-04-22 14:53:56,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:53:56,744 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1912): bcf49cb18a7738ad1232f3887a8bd9af/cf1 is initiating minor compaction (all files) 2023-04-22 14:53:56,745 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of bcf49cb18a7738ad1232f3887a8bd9af/cf1 in testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:56,746 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175229040 2023-04-22 14:53:56,746 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42-top] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/.tmp, totalSize=34.9 K 2023-04-22 14:53:56,747 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175229298 2023-04-22 14:53:56,747 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175229040 2023-04-22 14:53:56,748 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175229598 2023-04-22 14:53:56,748 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175229298 2023-04-22 14:53:56,748 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:56,748 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:56,748 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=bcf49cb18a7738ad1232f3887a8bd9af, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:56,749 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175236748"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175236748"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175236748"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175236748"}]},"ts":"1682175236748"} 2023-04-22 14:53:56,752 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682175229912 2023-04-22 14:53:56,752 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175229598 2023-04-22 14:53:56,753 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682175230309 2023-04-22 14:53:56,753 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682175229912 2023-04-22 14:53:56,754 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=32 2023-04-22 14:53:56,754 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=32, state=SUCCESS; OpenRegionProcedure bcf49cb18a7738ad1232f3887a8bd9af, server=jenkins-hbase4.apache.org,36267,1682175218583 in 560 msec 2023-04-22 14:53:56,754 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1682175230655 2023-04-22 14:53:56,755 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682175230309 2023-04-22 14:53:56,755 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175230968 2023-04-22 14:53:56,756 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1682175230655 2023-04-22 14:53:56,758 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175230968 2023-04-22 14:53:56,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=28 2023-04-22 14:53:56,763 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=bcf49cb18a7738ad1232f3887a8bd9af, ASSIGN in 731 msec 2023-04-22 14:53:56,777 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af in 5.5220 sec 2023-04-22 14:53:56,791 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] throttle.PressureAwareThroughputController(145): bcf49cb18a7738ad1232f3887a8bd9af#cf2#compaction#34 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:56,809 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] throttle.PressureAwareThroughputController(145): bcf49cb18a7738ad1232f3887a8bd9af#cf1#compaction#35 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:53:56,857 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/.tmp/cf2/098eb94d662d46e2af2b5b0f64c39e7b as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/098eb94d662d46e2af2b5b0f64c39e7b 2023-04-22 14:53:56,875 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in bcf49cb18a7738ad1232f3887a8bd9af/cf2 of bcf49cb18a7738ad1232f3887a8bd9af into 098eb94d662d46e2af2b5b0f64c39e7b(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:56,875 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcf49cb18a7738ad1232f3887a8bd9af: 2023-04-22 14:53:56,875 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af., storeName=bcf49cb18a7738ad1232f3887a8bd9af/cf2, priority=9, startTime=1682175236744; duration=0sec 2023-04-22 14:53:56,875 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:56,896 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/.tmp/cf1/8e8ec9256b7545c4865269c908b21c56 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/8e8ec9256b7545c4865269c908b21c56 2023-04-22 14:53:56,903 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in bcf49cb18a7738ad1232f3887a8bd9af/cf1 of bcf49cb18a7738ad1232f3887a8bd9af into 8e8ec9256b7545c4865269c908b21c56(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:53:56,903 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for bcf49cb18a7738ad1232f3887a8bd9af: 2023-04-22 14:53:56,903 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af., storeName=bcf49cb18a7738ad1232f3887a8bd9af/cf1, priority=9, startTime=1682175236743; duration=0sec 2023-04-22 14:53:56,904 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:53:57,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36267] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Scan size: 143 connection: 172.31.14.131:50214 deadline: 1682175297006, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84. is not online on jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:57,133 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-22 14:53:57,155 INFO [Listener at localhost/43403] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-04-22 14:53:57,160 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-22 14:53:57,209 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:53:57,211 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testSplitWithoutPONR 2023-04-22 14:53:57,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitWithoutPONR 2023-04-22 14:53:57,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:57,220 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175237220"}]},"ts":"1682175237220"} 2023-04-22 14:53:57,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 14:53:57,222 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLING in hbase:meta 2023-04-22 14:53:57,225 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitWithoutPONR to state=DISABLING 2023-04-22 14:53:57,228 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a8630043203581c85952b1c770f58cd9, UNASSIGN}, {pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=bcf49cb18a7738ad1232f3887a8bd9af, UNASSIGN}] 2023-04-22 14:53:57,229 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=bcf49cb18a7738ad1232f3887a8bd9af, UNASSIGN 2023-04-22 14:53:57,231 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=bcf49cb18a7738ad1232f3887a8bd9af, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:57,231 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175237231"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175237231"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175237231"}]},"ts":"1682175237231"} 2023-04-22 14:53:57,234 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=38, ppid=37, state=RUNNABLE; CloseRegionProcedure bcf49cb18a7738ad1232f3887a8bd9af, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:57,241 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a8630043203581c85952b1c770f58cd9, UNASSIGN 2023-04-22 14:53:57,242 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a8630043203581c85952b1c770f58cd9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:57,243 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175237242"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175237242"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175237242"}]},"ts":"1682175237242"} 2023-04-22 14:53:57,245 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=36, state=RUNNABLE; CloseRegionProcedure a8630043203581c85952b1c770f58cd9, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:53:57,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 14:53:57,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:57,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing bcf49cb18a7738ad1232f3887a8bd9af, disabling compactions & flushes 2023-04-22 14:53:57,387 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:57,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:57,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. after waiting 0 ms 2023-04-22 14:53:57,387 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:57,409 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42-top] to archive 2023-04-22 14:53:57,413 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:53:57,416 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,418 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,419 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,421 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,423 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,425 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,427 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,429 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f-top] to archive 2023-04-22 14:53:57,432 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:53:57,434 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,436 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,437 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,439 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,441 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,443 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,444 DEBUG [StoreCloser-testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:53:57,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:53:57,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af. 2023-04-22 14:53:57,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for bcf49cb18a7738ad1232f3887a8bd9af: 2023-04-22 14:53:57,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:57,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:57,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a8630043203581c85952b1c770f58cd9, disabling compactions & flushes 2023-04-22 14:53:57,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:57,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:57,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. after waiting 0 ms 2023-04-22 14:53:57,453 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:57,453 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=bcf49cb18a7738ad1232f3887a8bd9af, regionState=CLOSED 2023-04-22 14:53:57,454 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175237453"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175237453"}]},"ts":"1682175237453"} 2023-04-22 14:53:57,455 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42-bottom] to archive 2023-04-22 14:53:57,456 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:53:57,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=38, resume processing ppid=37 2023-04-22 14:53:57,459 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=37, state=SUCCESS; CloseRegionProcedure bcf49cb18a7738ad1232f3887a8bd9af, server=jenkins-hbase4.apache.org,36267,1682175218583 in 222 msec 2023-04-22 14:53:57,459 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,461 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=bcf49cb18a7738ad1232f3887a8bd9af, UNASSIGN in 231 msec 2023-04-22 14:53:57,464 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,466 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,470 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,472 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,477 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,479 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,482 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f-bottom] to archive 2023-04-22 14:53:57,483 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:53:57,486 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,488 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,490 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,491 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,493 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,496 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,498 DEBUG [StoreCloser-testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:53:57,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:53:57,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9. 2023-04-22 14:53:57,503 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a8630043203581c85952b1c770f58cd9: 2023-04-22 14:53:57,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:57,507 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a8630043203581c85952b1c770f58cd9, regionState=CLOSED 2023-04-22 14:53:57,507 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175237506"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175237506"}]},"ts":"1682175237506"} 2023-04-22 14:53:57,511 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=36 2023-04-22 14:53:57,511 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; CloseRegionProcedure a8630043203581c85952b1c770f58cd9, server=jenkins-hbase4.apache.org,36267,1682175218583 in 264 msec 2023-04-22 14:53:57,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-22 14:53:57,515 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=testSplitWithoutPONR, region=a8630043203581c85952b1c770f58cd9, UNASSIGN in 283 msec 2023-04-22 14:53:57,516 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175237516"}]},"ts":"1682175237516"} 2023-04-22 14:53:57,518 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitWithoutPONR, state=DISABLED in hbase:meta 2023-04-22 14:53:57,520 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitWithoutPONR to state=DISABLED 2023-04-22 14:53:57,523 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR in 309 msec 2023-04-22 14:53:57,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-22 14:53:57,525 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitWithoutPONR, procId: 35 completed 2023-04-22 14:53:57,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitWithoutPONR 2023-04-22 14:53:57,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:57,529 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=40, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:57,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-04-22 14:53:57,531 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=40, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:57,540 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,548 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:57,548 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:57,551 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/recovered.edits] 2023-04-22 14:53:57,552 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/recovered.edits] 2023-04-22 14:53:57,556 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/recovered.edits] 2023-04-22 14:53:57,560 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/.regioninfo to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/.regioninfo 2023-04-22 14:53:57,561 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/0d39e311ab844856b36212e9324b5524 2023-04-22 14:53:57,564 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3c4da0b13cc04b9ba7dbedf28988cf42 2023-04-22 14:53:57,569 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/3d63774c5ca5488eb4a7706eb2f81de8 2023-04-22 14:53:57,569 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,569 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/.regioninfo to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/.regioninfo 2023-04-22 14:53:57,571 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/5d3038915c9d4e879e6f1c48bd070ad7 2023-04-22 14:53:57,571 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,577 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8 2023-04-22 14:53:57,577 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,577 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/0d39e311ab844856b36212e9324b5524.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,580 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,581 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/3c4da0b13cc04b9ba7dbedf28988cf42.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,581 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/b95bdf6639764f25b10d00c627ed4a7a 2023-04-22 14:53:57,582 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,583 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf1/ff7154718348403b81f4561a26fc586e 2023-04-22 14:53:57,583 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/3d63774c5ca5488eb4a7706eb2f81de8.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,584 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,587 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/5d3038915c9d4e879e6f1c48bd070ad7.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,588 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,589 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/1ccccd66690041609009da38872649da 2023-04-22 14:53:57,590 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/af1cb7c12ae24af6b1d1d7ae4b04b0e8.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,593 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/2c9c93d70ecf49ac95c488011ed60e24 2023-04-22 14:53:57,593 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/b95bdf6639764f25b10d00c627ed4a7a.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,595 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,598 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,599 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf1/ff7154718348403b81f4561a26fc586e.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,601 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/afd8aad850e34c97bdae852fd2a0c6be 2023-04-22 14:53:57,603 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,607 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/c013e8636d7b46fe951dbbfe0bf01f23 2023-04-22 14:53:57,607 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,608 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/1ccccd66690041609009da38872649da.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,611 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,612 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/d4940bb491214ece9cbdb8f596d7a186 2023-04-22 14:53:57,613 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,614 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/2c9c93d70ecf49ac95c488011ed60e24.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,616 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/ec34b2adc15d4790a2210b4a2ad3a096 2023-04-22 14:53:57,618 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/bcf49cb18a7738ad1232f3887a8bd9af/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,620 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/cf2/fd5e75f6a08441c08e84e47e72cfc20f 2023-04-22 14:53:57,621 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/afd8aad850e34c97bdae852fd2a0c6be.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,625 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/c013e8636d7b46fe951dbbfe0bf01f23.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,627 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/8e8ec9256b7545c4865269c908b21c56 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf1/8e8ec9256b7545c4865269c908b21c56 2023-04-22 14:53:57,631 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/d4940bb491214ece9cbdb8f596d7a186.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,631 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/recovered.edits/85.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84/recovered.edits/85.seqid 2023-04-22 14:53:57,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-04-22 14:53:57,634 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/ec34b2adc15d4790a2210b4a2ad3a096.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,635 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,638 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/098eb94d662d46e2af2b5b0f64c39e7b to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/cf2/098eb94d662d46e2af2b5b0f64c39e7b 2023-04-22 14:53:57,639 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/a8630043203581c85952b1c770f58cd9/cf2/fd5e75f6a08441c08e84e47e72cfc20f.b8acaab15ee577f218a114962304cb84 2023-04-22 14:53:57,652 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af/recovered.edits/90.seqid 2023-04-22 14:53:57,658 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:53:57,664 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/a5a6de66b1324674aa4350248fc88ee1 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf1/a5a6de66b1324674aa4350248fc88ee1 2023-04-22 14:53:57,669 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/10a4504351a34451a127ba8395850699 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/cf2/10a4504351a34451a127ba8395850699 2023-04-22 14:53:57,677 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9/recovered.edits/90.seqid 2023-04-22 14:53:57,677 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitWithoutPONR/a8630043203581c85952b1c770f58cd9 2023-04-22 14:53:57,677 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitWithoutPONR regions 2023-04-22 14:53:57,681 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=40, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:57,683 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitWithoutPONR from hbase:meta 2023-04-22 14:53:57,687 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitWithoutPONR' descriptor. 2023-04-22 14:53:57,688 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=40, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:57,689 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitWithoutPONR' from region states. 2023-04-22 14:53:57,689 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175237689"}]},"ts":"9223372036854775807"} 2023-04-22 14:53:57,689 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175237689"}]},"ts":"9223372036854775807"} 2023-04-22 14:53:57,689 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175237689"}]},"ts":"9223372036854775807"} 2023-04-22 14:53:57,691 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-22 14:53:57,691 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b8acaab15ee577f218a114962304cb84, NAME => 'testSplitWithoutPONR,,1682175228283.b8acaab15ee577f218a114962304cb84.', STARTKEY => '', ENDKEY => ''}, {ENCODED => a8630043203581c85952b1c770f58cd9, NAME => 'testSplitWithoutPONR,,1682175231241.a8630043203581c85952b1c770f58cd9.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => bcf49cb18a7738ad1232f3887a8bd9af, NAME => 'testSplitWithoutPONR,41,1682175231241.bcf49cb18a7738ad1232f3887a8bd9af.', STARTKEY => '41', ENDKEY => ''}] 2023-04-22 14:53:57,692 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitWithoutPONR' as deleted. 2023-04-22 14:53:57,692 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitWithoutPONR","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175237692"}]},"ts":"9223372036854775807"} 2023-04-22 14:53:57,694 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitWithoutPONR state from META 2023-04-22 14:53:57,696 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=40, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:53:57,698 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR in 170 msec 2023-04-22 14:53:57,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=40 2023-04-22 14:53:57,834 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitWithoutPONR, procId: 40 completed 2023-04-22 14:53:57,851 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitWithoutPONR Thread=398 (was 392) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:35716 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:35762 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:58526 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:51632 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:58476 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:50110 [Waiting for operation #38] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:35696 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:35702 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:35278 [Waiting for operation #17] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:50032 [Waiting for operation #11] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:50096 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:54638 [Waiting for operation #37] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:49832 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:40048 [Waiting for operation #28] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:35280 [Waiting for operation #8] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x42387a87-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1298617065_15 at /127.0.0.1:40066 [Waiting for operation #27] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=846 (was 820) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=428 (was 354) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=12676 (was 12978) 2023-04-22 14:53:57,873 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=398, OpenFileDescriptor=846, MaxFileDescriptor=60000, SystemLoadAverage=428, ProcessCount=172, AvailableMemoryMB=12676 2023-04-22 14:53:57,873 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:53:57,875 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:53:57,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:53:57,878 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:53:57,880 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:53:57,881 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:53:57,883 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:57,883 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc empty. 2023-04-22 14:53:57,884 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:57,884 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-04-22 14:53:57,898 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-04-22 14:53:57,900 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(7675): creating {ENCODED => 68494e94cc440349573fb09884f7f6bc, NAME => 'testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:53:57,918 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(866): Instantiated testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:57,918 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1604): Closing 68494e94cc440349573fb09884f7f6bc, disabling compactions & flushes 2023-04-22 14:53:57,918 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1626): Closing region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:57,918 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:57,918 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1714): Acquired close lock on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. after waiting 0 ms 2023-04-22 14:53:57,918 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1724): Updates disabled for region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:57,919 INFO [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1838): Closed testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:57,919 DEBUG [RegionOpenAndInit-testRollbackAndDoubleExecution-pool-0] regionserver.HRegion(1558): Region close journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:53:57,924 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:53:57,926 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175237926"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175237926"}]},"ts":"1682175237926"} 2023-04-22 14:53:57,928 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:53:57,929 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:53:57,930 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175237929"}]},"ts":"1682175237929"} 2023-04-22 14:53:57,932 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLING in hbase:meta 2023-04-22 14:53:57,936 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:53:57,938 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:53:57,938 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:53:57,938 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:53:57,938 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:53:57,938 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN}] 2023-04-22 14:53:57,941 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:53:57,942 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=41, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35417,1682175218529; forceNewPlan=false, retain=false 2023-04-22 14:53:58,092 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:53:58,094 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,094 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175238094"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175238094"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175238094"}]},"ts":"1682175238094"} 2023-04-22 14:53:58,097 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=42, state=RUNNABLE; OpenRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:53:58,119 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:53:58,258 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:58,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 68494e94cc440349573fb09884f7f6bc, NAME => 'testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:53:58,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:53:58,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:58,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:53:58,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:58,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:58,264 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:58,266 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1 2023-04-22 14:53:58,266 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1 2023-04-22 14:53:58,266 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 68494e94cc440349573fb09884f7f6bc columnFamilyName cf1 2023-04-22 14:53:58,267 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(310): Store=68494e94cc440349573fb09884f7f6bc/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:58,267 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:58,268 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2 2023-04-22 14:53:58,268 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2 2023-04-22 14:53:58,268 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 68494e94cc440349573fb09884f7f6bc columnFamilyName cf2 2023-04-22 14:53:58,269 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(310): Store=68494e94cc440349573fb09884f7f6bc/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:53:58,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:58,270 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:58,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:53:58,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:53:58,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:53:58,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 68494e94cc440349573fb09884f7f6bc; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71622447, jitterRate=0.06725762784481049}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:53:58,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:53:58,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc., pid=43, masterSystemTime=1682175238249 2023-04-22 14:53:58,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:58,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:58,280 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,280 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175238280"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175238280"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175238280"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175238280"}]},"ts":"1682175238280"} 2023-04-22 14:53:58,284 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=42 2023-04-22 14:53:58,284 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=42, state=SUCCESS; OpenRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 in 185 msec 2023-04-22 14:53:58,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=41 2023-04-22 14:53:58,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=41, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN in 346 msec 2023-04-22 14:53:58,287 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:53:58,287 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175238287"}]},"ts":"1682175238287"} 2023-04-22 14:53:58,289 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackAndDoubleExecution' 2023-04-22 14:53:58,290 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=ENABLED in hbase:meta 2023-04-22 14:53:58,303 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=41, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:53:58,306 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution in 426 msec 2023-04-22 14:53:58,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:58,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-22 14:53:58,392 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-22 14:53:58,392 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,392 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-22 14:53:58,392 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:53:58,393 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,393 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,394 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,394 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,394 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,394 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,395 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,395 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:58,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:58,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:58,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:58,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:58,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:58,397 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:58,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:58,398 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:58,399 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,400 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:58,400 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:58,400 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:58,400 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. started... 2023-04-22 14:53:58,401 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 68494e94cc440349573fb09884f7f6bc 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-22 14:53:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,401 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,402 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,418 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/a55a3c6d837b439aac7d739c3b624761 2023-04-22 14:53:58,439 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/3533885279e04f4a9dfabbab87abf200 2023-04-22 14:53:58,445 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/a55a3c6d837b439aac7d739c3b624761 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761 2023-04-22 14:53:58,450 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:53:58,451 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/3533885279e04f4a9dfabbab87abf200 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200 2023-04-22 14:53:58,457 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:53:58,458 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 68494e94cc440349573fb09884f7f6bc in 57ms, sequenceid=6, compaction requested=false 2023-04-22 14:53:58,459 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:53:58,459 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:58,459 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:58,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,461 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,462 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,462 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-22 14:53:58,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@643f914e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:58,466 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:53:58,466 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:58,468 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,468 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:58,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:58,469 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:58,469 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:58,471 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,471 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,473 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:58,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,35417,1682175218529': 2023-04-22 14:53:58,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:53:58,476 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-22 14:53:58,476 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:58,476 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:58,476 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-22 14:53:58,477 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:58,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,480 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,480 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,480 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,483 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,483 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,483 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,487 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,487 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,487 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,487 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,487 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,487 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,488 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-22 14:53:58,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,489 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:58,489 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:58,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution because node does not exist (not an error) 2023-04-22 14:53:58,589 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:58,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:58,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:58,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:58,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-22 14:53:58,615 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-22 14:53:58,615 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,615 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-22 14:53:58,616 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:53:58,616 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,616 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,618 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,618 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,618 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,619 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,620 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,620 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:58,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:58,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:58,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:58,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,622 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:58,623 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:58,625 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,626 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:58,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,629 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:58,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:58,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:58,631 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:58,631 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:58,631 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. started... 2023-04-22 14:53:58,632 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 68494e94cc440349573fb09884f7f6bc 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,633 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,634 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,685 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/17172f828c0e4f8d8e82ad29a1c72456 2023-04-22 14:53:58,712 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/b12cee7be67d466cbe1457f8bc4723f0 2023-04-22 14:53:58,720 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/17172f828c0e4f8d8e82ad29a1c72456 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456 2023-04-22 14:53:58,725 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:53:58,727 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/b12cee7be67d466cbe1457f8bc4723f0 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0 2023-04-22 14:53:58,733 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:53:58,735 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 68494e94cc440349573fb09884f7f6bc in 103ms, sequenceid=19, compaction requested=false 2023-04-22 14:53:58,735 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:53:58,735 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:58,735 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:58,737 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,737 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,738 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-22 14:53:58,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f1a810f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:58,742 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:53:58,742 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,745 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,745 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,745 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:58,747 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,747 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,748 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:58,750 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:58,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,750 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:58,751 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:58,751 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,751 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,35417,1682175218529': 2023-04-22 14:53:58,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:53:58,758 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-22 14:53:58,758 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:58,758 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:58,758 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-22 14:53:58,758 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:58,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,760 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,761 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,761 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,761 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,762 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,764 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,765 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,767 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,768 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,768 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,771 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,771 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,771 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,771 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,771 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,772 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:58,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,772 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,772 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,773 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-22 14:53:58,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:58,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:58,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:58,773 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:58,874 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:58,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:58,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:58,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:58,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-22 14:53:58,909 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-22 14:53:58,909 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,909 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-22 14:53:58,909 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:53:58,909 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,909 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,911 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,911 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,911 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,911 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:58,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,911 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,911 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:58,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:58,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:58,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:58,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:58,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:58,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:58,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:58,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:58,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:58,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:58,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:58,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,913 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:58,914 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. started... 2023-04-22 14:53:58,913 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:58,914 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:58,914 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:58,914 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 68494e94cc440349573fb09884f7f6bc 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:58,914 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:58,914 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:58,914 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:58,915 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,916 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,917 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,938 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/df2c1a7be9ed4995b2d8b34082133c1c 2023-04-22 14:53:58,956 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/ac35848cc24d423e8894a0aea7589d54 2023-04-22 14:53:58,968 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/df2c1a7be9ed4995b2d8b34082133c1c as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c 2023-04-22 14:53:58,979 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:53:58,981 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/ac35848cc24d423e8894a0aea7589d54 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54 2023-04-22 14:53:58,986 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:53:58,987 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 68494e94cc440349573fb09884f7f6bc in 73ms, sequenceid=32, compaction requested=true 2023-04-22 14:53:58,987 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:53:58,987 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:58,987 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:58,987 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:58,987 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,987 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:58,987 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:58,991 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:58,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-22 14:53:58,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16ba02b4[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:58,994 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:53:58,994 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,995 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,995 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:58,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,995 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:58,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:58,995 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,996 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:58,998 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:58,999 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,35417,1682175218529': 2023-04-22 14:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:53:59,006 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-22 14:53:59,006 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:59,006 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:59,006 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-22 14:53:59,006 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:59,013 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,013 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,014 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,013 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,013 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,014 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,014 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,016 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,017 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,017 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,019 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,020 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,020 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-22 14:53:59,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:59,029 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,029 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,030 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:59,030 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:59,030 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,030 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,030 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,030 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,030 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,030 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,030 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,130 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:59,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:59,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:59,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:59,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-22 14:53:59,158 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-22 14:53:59,158 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,158 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-22 14:53:59,158 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:53:59,158 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,159 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,160 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,160 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,160 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,160 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,161 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,161 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:59,162 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,163 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:59,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:59,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:59,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:59,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,164 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:59,164 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. started... 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:59,164 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 68494e94cc440349573fb09884f7f6bc 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:59,164 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:59,166 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,168 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,168 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,168 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,199 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/11c59ba427164756ae1d366488ee2d17 2023-04-22 14:53:59,242 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/86caaf5465a24975808612073073257a 2023-04-22 14:53:59,255 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/11c59ba427164756ae1d366488ee2d17 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17 2023-04-22 14:53:59,262 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:53:59,263 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/86caaf5465a24975808612073073257a as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a 2023-04-22 14:53:59,269 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:53:59,272 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 68494e94cc440349573fb09884f7f6bc in 108ms, sequenceid=45, compaction requested=true 2023-04-22 14:53:59,272 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:53:59,272 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:59,272 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:59,273 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:59,273 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,273 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:59,273 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,275 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-22 14:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6808e173[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:59,277 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:53:59,278 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:59,279 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,279 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:59,280 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:59,280 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:59,280 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,283 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,284 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,35417,1682175218529': 2023-04-22 14:53:59,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:53:59,292 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-22 14:53:59,292 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:59,292 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:59,292 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-22 14:53:59,292 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:59,294 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,294 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,294 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,295 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,295 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,295 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,295 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,295 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,295 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,296 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,298 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,298 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,299 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,302 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,302 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,302 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,302 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,303 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-22 14:53:59,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:59,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:59,303 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,304 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,304 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,304 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,304 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,304 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,404 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:59,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:59,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:59,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:59,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-22 14:53:59,424 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-22 14:53:59,424 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,424 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-22 14:53:59,424 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:53:59,424 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,424 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,426 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,426 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,426 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,426 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,427 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,427 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:59,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:59,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,428 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:59,429 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,429 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:59,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:59,429 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:59,429 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:59,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:59,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:59,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:59,430 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:59,430 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. started... 2023-04-22 14:53:59,430 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 68494e94cc440349573fb09884f7f6bc 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:59,430 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,432 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,432 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,432 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,432 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,432 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,445 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/a30570688f314130ab142b7829fe562f 2023-04-22 14:53:59,472 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/9437fc31aa604e9490bd2fdbf9b59292 2023-04-22 14:53:59,477 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/a30570688f314130ab142b7829fe562f as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f 2023-04-22 14:53:59,483 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:53:59,484 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/9437fc31aa604e9490bd2fdbf9b59292 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292 2023-04-22 14:53:59,489 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:53:59,490 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 68494e94cc440349573fb09884f7f6bc in 60ms, sequenceid=58, compaction requested=true 2023-04-22 14:53:59,491 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:53:59,491 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:59,491 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:59,491 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:59,491 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,491 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:59,491 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:59,493 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,493 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,493 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,493 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-22 14:53:59,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7025a8dd[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:59,496 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:53:59,496 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,499 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:59,499 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,499 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:59,501 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,501 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,501 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,508 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,508 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:59,508 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:59,508 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:59,508 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,508 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:59,509 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:59,512 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,512 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,514 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,515 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,35417,1682175218529': 2023-04-22 14:53:59,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:53:59,519 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-22 14:53:59,519 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:59,519 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:59,519 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-22 14:53:59,519 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:59,521 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,521 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,522 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,523 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,523 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,525 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,526 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,526 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,531 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,531 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,531 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,531 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,531 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,531 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,531 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,531 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,532 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,533 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,533 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-22 14:53:59,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:59,534 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:59,534 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:59,634 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:59,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:53:59,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:53:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRollbackAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:53:59,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRollbackAndDoubleExecution 2023-04-22 14:53:59,660 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRollbackAndDoubleExecution' 2023-04-22 14:53:59,660 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,660 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRollbackAndDoubleExecution' starting 'acquire' 2023-04-22 14:53:59,660 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRollbackAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:53:59,660 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,660 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,663 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,664 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,664 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:59,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:59,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRollbackAndDoubleExecution 2023-04-22 14:53:59,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:59,665 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:59,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRollbackAndDoubleExecution is still present, removing 2023-04-22 14:53:59,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRollbackAndDoubleExecution 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRollbackAndDoubleExecution' with timeout 60000ms 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRollbackAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:53:59,667 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:59,667 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. started... 2023-04-22 14:53:59,667 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 68494e94cc440349573fb09884f7f6bc 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:53:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,668 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,689 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/31eeb60011a54db1aa76957e8575c767 2023-04-22 14:53:59,712 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/5e946366082547e6b5750eceb6a14f2a 2023-04-22 14:53:59,718 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/31eeb60011a54db1aa76957e8575c767 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767 2023-04-22 14:53:59,723 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:53:59,725 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/5e946366082547e6b5750eceb6a14f2a as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a 2023-04-22 14:53:59,730 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:53:59,731 INFO [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 68494e94cc440349573fb09884f7f6bc in 64ms, sequenceid=71, compaction requested=true 2023-04-22 14:53:59,731 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:53:59,731 DEBUG [rs(jenkins-hbase4.apache.org,35417,1682175218529)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRollbackAndDoubleExecution' locally acquired 2023-04-22 14:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRollbackAndDoubleExecution) in zk 2023-04-22 14:53:59,733 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,733 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,734 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,734 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRollbackAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure 'testRollbackAndDoubleExecution' on coordinator 2023-04-22 14:53:59,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b1c4d2f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:53:59,737 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRollbackAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:53:59,737 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,738 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,738 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,738 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:59,738 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:59,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,738 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:53:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:59,739 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:53:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,739 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,739 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:53:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRollbackAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:53:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRollbackAndDoubleExecution' locally completed 2023-04-22 14:53:59,739 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRollbackAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:53:59,742 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRollbackAndDoubleExecution' has notified controller of completion 2023-04-22 14:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRollbackAndDoubleExecution' completed. 2023-04-22 14:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRollbackAndDoubleExecution' member 'jenkins-hbase4.apache.org,35417,1682175218529': 2023-04-22 14:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' released barrier for procedure'testRollbackAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:53:59,748 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRollbackAndDoubleExecution' execution completed 2023-04-22 14:53:59,748 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:53:59,748 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:53:59,748 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRollbackAndDoubleExecution 2023-04-22 14:53:59,748 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRollbackAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:53:59,750 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,750 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,750 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,750 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,750 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,751 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,751 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,752 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,753 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:53:59,754 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:53:59,755 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRollbackAndDoubleExecution 2023-04-22 14:53:59,755 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,756 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:53:59,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:53:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,765 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:53:59,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRollbackAndDoubleExecution' 2023-04-22 14:53:59,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:53:59,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRollbackAndDoubleExecution 2023-04-22 14:53:59,765 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRollbackAndDoubleExecution 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRollbackAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:53:59,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,766 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRollbackAndDoubleExecution 2023-04-22 14:53:59,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:53:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:53:59,866 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:53:59,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:00,136 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-04-22 14:54:00,137 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:00,138 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,139 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,139 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,139 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,141 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:00,141 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,141 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:00,141 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:00,388 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 2 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,389 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:00,389 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:00,389 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:00,389 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-04-22 14:54:00,389 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:00,389 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:00,389 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:00,390 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:00,390 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:00,391 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-04-22 14:54:00,392 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:00,392 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:00,393 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:00,393 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:00,393 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:00,393 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:00,394 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:00,394 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:00,394 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:00,394 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:00,395 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:00,395 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:00,395 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:00,395 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,395 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:00,396 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,396 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,396 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,396 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-22 14:54:00,397 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:00,400 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:00,400 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175270400 2023-04-22 14:54:00,400 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:00,402 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,405 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:00,406 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:00,406 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:00,406 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175300406 2023-04-22 14:54:00,406 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175360406 2023-04-22 14:54:00,406 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-22 14:54:00,406 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:00,408 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:00,408 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,408 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:00,409 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:00,656 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 3 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,657 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:00,657 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:00,657 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:00,660 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:00,660 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:00,660 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:00,660 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:00,660 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:00,661 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:00,662 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-04-22 14:54:00,663 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:00,663 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:00,664 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:00,664 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:00,664 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:00,665 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:00,665 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:00,665 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:00,666 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:00,666 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:00,666 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:00,667 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:00,667 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:00,667 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,667 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:00,668 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,668 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,668 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,668 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-22 14:54:00,669 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:00,669 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:00,671 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175270671 2023-04-22 14:54:00,672 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:00,673 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,675 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:00,676 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:00,676 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:00,676 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175300676 2023-04-22 14:54:00,676 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175360676 2023-04-22 14:54:00,676 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:00,676 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:00,680 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN}] 2023-04-22 14:54:00,681 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:00,681 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,681 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:00,681 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:00,926 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,926 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:00,927 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:00,927 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:00,927 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:00,927 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:00,927 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:00,927 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:00,927 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:00,928 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:00,929 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-04-22 14:54:00,929 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:00,930 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:00,930 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:00,930 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:00,930 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:00,931 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:00,931 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:00,931 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:00,932 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:00,932 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:00,933 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:00,933 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:00,934 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:00,934 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,934 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:00,934 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,934 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,935 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,935 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-22 14:54:00,935 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:00,936 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:00,942 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175270942 2023-04-22 14:54:00,945 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:00,945 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:00,947 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:00,948 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:00,948 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:00,948 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175300948 2023-04-22 14:54:00,948 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175360948 2023-04-22 14:54:00,948 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:00,949 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN}] 2023-04-22 14:54:00,949 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:00,951 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:00,952 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:00,952 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175240952"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175240952"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175240952"}]},"ts":"1682175240952"} 2023-04-22 14:54:00,954 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:54:00,954 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:00,954 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:00,954 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:00,954 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:01,198 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,199 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:01,199 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:01,199 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:01,199 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:01,199 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:01,199 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:01,199 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:01,200 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:01,201 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-22 14:54:01,202 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:54:01,202 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-22 14:54:01,202 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:01,203 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:01,203 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:01,203 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:01,204 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:01,206 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:01,206 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:01,206 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:01,207 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:01,207 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:01,207 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:01,207 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:01,208 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:01,209 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,209 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:01,209 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:01,209 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,209 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,209 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,209 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:01,210 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:01,210 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-04-22 14:54:01,218 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:01,218 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:01,218 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc to restore RIT 2023-04-22 14:54:01,219 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175271219 2023-04-22 14:54:01,220 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:01,220 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:01,223 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:01,223 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSING, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:01,223 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:01,223 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175301223 2023-04-22 14:54:01,223 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175361223 2023-04-22 14:54:01,224 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:01,224 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:01,224 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175241224"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175241224"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175241224"}]},"ts":"1682175241224"} 2023-04-22 14:54:01,226 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:54:01,226 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:01,402 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:01,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 68494e94cc440349573fb09884f7f6bc, disabling compactions & flushes 2023-04-22 14:54:01,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:01,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:01,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. after waiting 0 ms 2023-04-22 14:54:01,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:01,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 68494e94cc440349573fb09884f7f6bc 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-22 14:54:01,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/6d189a279acd4fd3874f0c48349dc663 2023-04-22 14:54:01,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/f23d308a83544ae3ac5a7790c861929a 2023-04-22 14:54:01,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/6d189a279acd4fd3874f0c48349dc663 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663 2023-04-22 14:54:01,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:54:01,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/f23d308a83544ae3ac5a7790c861929a as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a 2023-04-22 14:54:01,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:54:01,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 68494e94cc440349573fb09884f7f6bc in 61ms, sequenceid=82, compaction requested=true 2023-04-22 14:54:01,484 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-22 14:54:01,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:54:01,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:01,485 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:54:01,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:01,489 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED 2023-04-22 14:54:01,490 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175241489"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175241489"}]},"ts":"1682175241489"} 2023-04-22 14:54:01,491 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:01,491 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:01,491 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:01,492 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=46, ppid=45, state=SUCCESS, locked=true; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:01,725 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,725 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:01,725 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:01,725 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:01,726 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:01,726 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:01,727 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:01,727 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:01,727 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:01,728 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:01,729 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:54:01,729 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-04-22 14:54:01,729 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4] 2023-04-22 14:54:01,730 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:01,730 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:01,730 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:01,731 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:01,731 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:01,731 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:01,731 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:01,732 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:01,732 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:01,732 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:01,733 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:01,733 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:01,733 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:01,734 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,734 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:01,734 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:01,734 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:01,734 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,734 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,734 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,735 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:01,735 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:01,735 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:01,735 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-22 14:54:01,736 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:01,736 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:01,736 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc to restore RIT 2023-04-22 14:54:01,737 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175271737 2023-04-22 14:54:01,738 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:01,738 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=46, ppid=45, state=RUNNABLE; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:01,741 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:01,741 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:01,741 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:01,741 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175301741 2023-04-22 14:54:01,741 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175361741 2023-04-22 14:54:01,741 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:01,742 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=45 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:01,742 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175241742"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175241742"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175241742"}]},"ts":"1682175241742"} 2023-04-22 14:54:01,743 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:01,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=45 2023-04-22 14:54:01,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 in 517 msec 2023-04-22 14:54:01,748 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:01,748 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:01,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:01,749 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=45, ppid=44, state=SUCCESS, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:01,992 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 11 exec state=pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:01,992 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:01,992 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:01,992 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:01,993 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:01,994 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:01,994 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:01,994 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:01,994 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:01,995 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:01,996 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:54:01,996 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3] 2023-04-22 14:54:01,996 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:54:01,997 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:01,997 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:01,998 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:01,998 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:01,998 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:01,999 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:01,999 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:01,999 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:01,999 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:02,000 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:02,000 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:02,000 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:02,001 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:02,002 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,002 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:02,002 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:02,002 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:02,002 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,002 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,002 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,003 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:02,003 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:02,003 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-04-22 14:54:02,003 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:02,004 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:02,004 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc to restore RIT 2023-04-22 14:54:02,004 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175272004 2023-04-22 14:54:02,005 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:02,006 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:02,008 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:02,008 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:02,008 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:02,008 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175302008 2023-04-22 14:54:02,008 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175362008 2023-04-22 14:54:02,008 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 2 msec 2023-04-22 14:54:02,008 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=45, ppid=44, state=RUNNABLE:REGION_STATE_TRANSITION_CONFIRM_CLOSED; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:02,011 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:02,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-22 14:54:02,015 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN in 1.0620 sec 2023-04-22 14:54:02,018 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:02,018 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,018 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:02,018 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:02,258 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 4 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,259 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:02,259 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:02,259 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:02,260 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:02,260 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:02,260 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:02,260 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:02,260 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:02,261 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:02,262 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:54:02,262 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:54:02,263 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:54:02,263 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:02,263 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:02,264 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:02,264 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:02,264 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:02,264 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:02,265 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:02,265 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:02,265 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:02,266 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:02,266 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:02,266 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:02,266 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:02,267 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,267 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:02,267 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:02,268 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:02,268 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:02,268 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:02,268 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-22 14:54:02,271 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:02,271 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:02,272 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175272272 2023-04-22 14:54:02,285 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,285 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:02,288 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:02,288 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:02,289 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:02,289 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175302289 2023-04-22 14:54:02,289 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175362289 2023-04-22 14:54:02,289 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:02,289 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,289 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,289 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:02,297 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:02,310 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=68494e94cc440349573fb09884f7f6bc, threads=14 2023-04-22 14:54:02,312 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,316 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,324 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,332 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,349 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,352 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,380 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,396 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,400 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,400 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,402 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,404 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,412 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,438 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,640 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,670 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,676 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,681 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,704 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,711 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,713 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,715 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,724 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,731 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:02,962 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,088 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,088 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,091 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,091 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region 68494e94cc440349573fb09884f7f6bc Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-22 14:54:03,166 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:03,166 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:03,166 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:03,166 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:03,289 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 5 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:03,290 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:03,290 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:03,290 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:03,291 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:03,291 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:03,291 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:03,291 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:03,291 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:03,292 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:03,294 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-04-22 14:54:03,294 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:54:03,294 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:54:03,294 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:03,295 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:03,295 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:03,295 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:03,296 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:03,296 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:03,296 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:03,296 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:03,297 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:03,297 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:03,297 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:03,297 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:03,298 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:03,298 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:03,299 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:03,299 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:03,299 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:03,299 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:03,299 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:03,299 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:03,299 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:03,299 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:03,299 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-22 14:54:03,300 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:03,301 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:03,301 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175273301 2023-04-22 14:54:03,303 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:03,303 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:03,306 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:03,306 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:03,306 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:03,306 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175303306 2023-04-22 14:54:03,306 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175363306 2023-04-22 14:54:03,306 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:03,323 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=44 splitting 14 storefiles, region=68494e94cc440349573fb09884f7f6bc, threads=14 2023-04-22 14:54:03,325 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,326 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,332 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,333 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,334 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,336 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,336 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,336 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,337 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,341 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,341 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,342 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,342 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,342 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=44 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,611 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,652 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,670 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,673 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,719 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,720 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,740 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,761 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:54:03,762 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,762 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,787 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,800 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,808 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,827 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,830 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=44 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663 for region: 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:03,830 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=44 split storefiles for region 68494e94cc440349573fb09884f7f6bc Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-22 14:54:03,876 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:03,882 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/4d7e9272d5568801647adcb548735695/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:54:03,884 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/36622fde6be694d7ba2dd620c7da97ed/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:54:03,884 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:03,884 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:03,884 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:03,884 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:04,057 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(417): Restart 6 exec state=pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,057 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:04,057 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:04,057 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:04,058 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:04,058 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:04,058 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:04,058 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:04,058 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:04,059 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:04,060 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-04-22 14:54:04,060 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:54:04,060 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:54:04,060 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:04,061 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:04,061 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:04,061 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:04,061 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,062 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:04,062 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,062 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,062 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,063 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:04,063 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,063 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,063 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:04,063 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,064 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:04,064 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:04,066 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:04,066 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,066 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,066 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,066 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:04,066 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:04,066 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 6 msec 2023-04-22 14:54:04,067 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:04,067 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:04,068 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175274068 2023-04-22 14:54:04,068 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:04,069 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,072 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:04,072 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:04,072 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:04,072 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175304072 2023-04-22 14:54:04,072 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175364072 2023-04-22 14:54:04,072 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:04,075 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:04,078 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:04,078 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:04,078 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:04,323 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(509): Restart 0 rollback state: pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,323 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:04,323 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:04,323 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:04,324 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:04,324 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:04,324 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:04,325 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:04,325 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:04,325 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:04,326 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-04-22 14:54:04,326 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:54:04,326 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:54:04,326 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:04,327 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:04,327 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:04,327 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:04,327 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,328 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:04,328 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,328 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,328 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,328 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:04,329 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,329 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,329 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:04,329 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,329 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:04,329 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:04,330 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:04,330 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,330 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,330 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,330 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:04,330 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:04,330 DEBUG [Listener at localhost/43403] procedure2.StateMachineProcedure(225): Abort requested for pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,330 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-04-22 14:54:04,331 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:04,331 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:04,332 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175274332 2023-04-22 14:54:04,332 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:04,334 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=44, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,338 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:04,338 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:04,338 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:04,338 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175304338 2023-04-22 14:54:04,338 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175364338 2023-04-22 14:54:04,338 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-04-22 14:54:04,339 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:04,352 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:04,352 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-22 14:54:04,352 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:04,352 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:04,590 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(509): Restart 1 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,590 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:04,590 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:04,590 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:04,590 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:04,590 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:04,590 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:04,590 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:04,590 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:04,591 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:04,592 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-04-22 14:54:04,592 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:54:04,593 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:54:04,593 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:04,593 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:04,593 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:04,593 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:04,594 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,594 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:04,594 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,594 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,594 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,595 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:04,595 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,595 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,595 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:04,596 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,596 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:04,596 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:04,596 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:04,596 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,597 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,597 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,597 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:04,597 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:04,597 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-22 14:54:04,597 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:04,598 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:04,598 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175274598 2023-04-22 14:54:04,600 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:04,601 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,603 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:04,603 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:04,603 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:04,603 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175304603 2023-04-22 14:54:04,604 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175364603 2023-04-22 14:54:04,604 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:04,604 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:04,606 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:04,606 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-22 14:54:04,606 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:04,606 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:04,855 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(509): Restart 2 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,855 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:04,855 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:04,855 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:04,855 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:04,855 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:04,855 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:04,855 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:04,855 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:04,856 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:04,857 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-04-22 14:54:04,857 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:54:04,857 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:54:04,858 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:04,858 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:04,858 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:04,858 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:04,859 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,859 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:04,859 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,859 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:04,859 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,860 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:04,860 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,860 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:04,860 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:04,861 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,861 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:04,861 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:04,861 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:04,861 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,861 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,861 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,862 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:04,862 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:04,862 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-22 14:54:04,862 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:04,863 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:04,863 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175274863 2023-04-22 14:54:04,864 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:04,865 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:04,867 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:04,867 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:04,867 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:04,868 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175304868 2023-04-22 14:54:04,868 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175364868 2023-04-22 14:54:04,868 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:04,868 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:04,870 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:04,870 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-22 14:54:04,870 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:04,871 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:05,118 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(509): Restart 3 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,118 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:05,118 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:05,118 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:05,119 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:05,119 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:05,119 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:05,119 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:05,119 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:05,120 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:05,121 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-04-22 14:54:05,121 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:54:05,121 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:54:05,121 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:05,122 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:05,122 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:05,122 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:05,123 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,123 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:05,123 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,123 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,123 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,124 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:05,124 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,124 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,124 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:05,125 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,125 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:05,125 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:05,125 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:05,125 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,126 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,126 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,126 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:05,126 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:05,126 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-22 14:54:05,127 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:05,127 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:05,127 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175275127 2023-04-22 14:54:05,128 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:05,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,132 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:05,132 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:05,132 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:05,132 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175305132 2023-04-22 14:54:05,132 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175365132 2023-04-22 14:54:05,132 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-22 14:54:05,132 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:05,136 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1054): Stored [pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN] 2023-04-22 14:54:05,136 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:05,136 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-22 14:54:05,136 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:05,136 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:05,382 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(509): Restart 4 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,383 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:05,383 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:05,383 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:05,385 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:05,385 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:05,385 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:05,385 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:05,386 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:05,386 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:05,387 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-04-22 14:54:05,387 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=45, ppid=44, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:54:05,387 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=46, ppid=45, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:54:05,387 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-22 14:54:05,387 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:05,388 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:05,388 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:05,388 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:05,388 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,389 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:05,389 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,389 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,389 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,389 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:05,389 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,390 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,390 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:05,391 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,391 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:05,391 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:05,391 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:05,391 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:05,391 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,391 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,391 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,391 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=45, ppid=44, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:05,391 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=46, ppid=45, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:05,391 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:05,391 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-04-22 14:54:05,392 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:05,392 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:05,393 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc to restore RIT 2023-04-22 14:54:05,393 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175275393 2023-04-22 14:54:05,394 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:05,395 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,396 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:05,397 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:05,398 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:05,398 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:05,398 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175305398 2023-04-22 14:54:05,398 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175365398 2023-04-22 14:54:05,398 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-22 14:54:05,398 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:05,402 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:05,403 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-22 14:54:05,403 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:05,403 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:05,648 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(509): Restart 5 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,648 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:05,648 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:05,648 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:05,649 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:05,649 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:05,649 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:05,649 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:05,649 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:05,650 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:05,650 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:54:05,650 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-22 14:54:05,651 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:05,651 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:05,651 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:05,651 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:05,651 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,652 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:05,652 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,652 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,652 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,652 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:05,652 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,653 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,653 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:05,653 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,653 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:05,653 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:05,653 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,653 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,654 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,654 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:05,654 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-04-22 14:54:05,654 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:05,655 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:05,655 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc to restore RIT 2023-04-22 14:54:05,655 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175275655 2023-04-22 14:54:05,657 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,657 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:05,658 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:05,660 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:05,660 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:05,661 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:05,661 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175305661 2023-04-22 14:54:05,661 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175365661 2023-04-22 14:54:05,661 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:05,661 INFO [PEWorker-1] procedure.MasterProcedureScheduler(718): Waiting on xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN held by pid=44 2023-04-22 14:54:05,661 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:05,661 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:05,663 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:05,663 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-22 14:54:05,663 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:05,663 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:05,911 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(509): Restart 6 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,911 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:05,911 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:05,911 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:05,912 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:05,912 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:05,912 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:05,912 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:05,912 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:05,913 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:05,914 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-04-22 14:54:05,914 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-22 14:54:05,914 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:05,915 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:05,915 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:05,915 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:05,915 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,916 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:05,916 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,916 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:05,916 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,917 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:05,917 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,917 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:05,917 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:05,918 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,918 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:05,918 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:05,918 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,918 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,918 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,918 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:05,919 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 5 msec 2023-04-22 14:54:05,919 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:05,920 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:05,920 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc to restore RIT 2023-04-22 14:54:05,920 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175275920 2023-04-22 14:54:05,921 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:05,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PRE_OPERATION, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:05,924 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:05,926 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:05,926 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:05,927 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:05,927 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175305927 2023-04-22 14:54:05,927 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175365927 2023-04-22 14:54:05,927 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-22 14:54:05,927 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:05,930 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:05,930 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1581): TESTING: Kill before store update 2023-04-22 14:54:05,930 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:05,930 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill before store update at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1583) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeRollback(ProcedureExecutor.java:1498) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1349) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:06,177 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(509): Restart 7 rollback state: pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, locked=true, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:06,177 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:06,177 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:06,178 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:06,178 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:06,178 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:06,178 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:06,178 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:06,178 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:06,179 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:06,179 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=44, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-04-22 14:54:06,180 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[] 2023-04-22 14:54:06,180 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:06,180 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:06,180 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:06,181 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:06,181 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:06,181 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:06,181 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:06,181 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:06,182 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:06,182 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:06,182 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:06,182 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:06,182 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:06,183 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:06,183 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:06,183 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:06,183 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:06,183 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:06,183 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:06,183 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:54:06,183 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-04-22 14:54:06,184 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:06,184 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:06,184 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc to restore RIT 2023-04-22 14:54:06,185 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175276185 2023-04-22 14:54:06,186 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:06,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1357): LOCK_EVENT_WAIT rollback...pid=44, state=FAILED:SPLIT_TABLE_REGION_PREPARE, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:06,189 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:06,191 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:06,191 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:06,191 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:06,191 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175306191 2023-04-22 14:54:06,191 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175366191 2023-04-22 14:54:06,191 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-22 14:54:06,191 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:54:06,193 INFO [PEWorker-1] procedure2.ProcedureExecutor(1528): Rolled back pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed exec-time=6.0540 sec 2023-04-22 14:54:06,193 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:06,194 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,35417,1682175218529; forceNewPlan=false, retain=true 2023-04-22 14:54:06,344 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:54:06,345 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:06,346 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175246345"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175246345"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175246345"}]},"ts":"1682175246345"} 2023-04-22 14:54:06,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:54:06,348 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:54:06,348 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:06,348 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:54:06,348 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=47, state=WAITING:REGION_STATE_TRANSITION_CONFIRM_OPENED, locked=true; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:54:06,441 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:54:06,441 WARN [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(526): ProcedureExecutor not running, may have been stopped by pending procedure due to KillAndToggleBeforeStoreUpdate flag. 2023-04-22 14:54:06,441 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:54:06,442 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:54:06,442 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:54:06,442 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:54:06,442 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:54:06,442 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:54:06,442 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:54:06,442 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:54:06,443 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:54:06,444 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=47, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[0] 2023-04-22 14:54:06,444 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:54:06,444 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:54:06,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:54:06,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:54:06,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:06,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:54:06,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:06,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:54:06,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:06,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:54:06,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:06,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:54:06,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:06,447 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:54:06,447 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:06,447 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:54:06,447 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:06,447 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 4 msec 2023-04-22 14:54:06,448 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:54:06,449 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:54:06,449 INFO [Listener at localhost/43403] assignment.AssignmentManager(305): Attach pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN to state=OFFLINE, location=null, table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc to restore RIT 2023-04-22 14:54:06,450 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175276450 2023-04-22 14:54:06,451 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=47, state=RUNNABLE:REGION_STATE_TRANSITION_OPEN; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:54:06,451 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:54:06,453 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:54:06,454 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=68494e94cc440349573fb09884f7f6bc, regionState=OPENING, lastHost=jenkins-hbase4.apache.org,35417,1682175218529, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529, openSeqNum=2 2023-04-22 14:54:06,454 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:54:06,454 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175306454 2023-04-22 14:54:06,454 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175366454 2023-04-22 14:54:06,454 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:54:06,454 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:06,454 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175246454"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175246454"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175246454"}]},"ts":"1682175246454"} 2023-04-22 14:54:06,456 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE; OpenRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:54:06,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:06,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 68494e94cc440349573fb09884f7f6bc, NAME => 'testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:54:06,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:06,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackAndDoubleExecution 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:06,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,614 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,615 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1 2023-04-22 14:54:06,615 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1 2023-04-22 14:54:06,615 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 68494e94cc440349573fb09884f7f6bc columnFamilyName cf1 2023-04-22 14:54:06,622 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17 2023-04-22 14:54:06,625 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456 2023-04-22 14:54:06,629 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767 2023-04-22 14:54:06,632 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663 2023-04-22 14:54:06,636 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f 2023-04-22 14:54:06,639 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761 2023-04-22 14:54:06,642 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c 2023-04-22 14:54:06,642 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(310): Store=68494e94cc440349573fb09884f7f6bc/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:06,643 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,644 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2 2023-04-22 14:54:06,644 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2 2023-04-22 14:54:06,644 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 68494e94cc440349573fb09884f7f6bc columnFamilyName cf2 2023-04-22 14:54:06,652 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200 2023-04-22 14:54:06,656 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a 2023-04-22 14:54:06,660 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a 2023-04-22 14:54:06,664 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292 2023-04-22 14:54:06,669 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54 2023-04-22 14:54:06,673 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0 2023-04-22 14:54:06,676 DEBUG [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a 2023-04-22 14:54:06,677 INFO [StoreOpener-68494e94cc440349573fb09884f7f6bc-1] regionserver.HStore(310): Store=68494e94cc440349573fb09884f7f6bc/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:06,677 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,678 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(602): Cleaned up old failed split transaction detritus: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.splits 2023-04-22 14:54:06,681 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRollbackAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:54:06,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,683 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 68494e94cc440349573fb09884f7f6bc; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61180673, jitterRate=-0.08833692967891693}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:54:06,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:54:06,684 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc., pid=48, masterSystemTime=1682175246609 2023-04-22 14:54:06,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:06,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:06,685 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:54:06,685 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:54:06,686 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 14:54:06,686 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 14:54:06,686 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] regionserver.HStore(1912): 68494e94cc440349573fb09884f7f6bc/cf1 is initiating minor compaction (all files) 2023-04-22 14:54:06,686 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] regionserver.HStore(1912): 68494e94cc440349573fb09884f7f6bc/cf2 is initiating minor compaction (all files) 2023-04-22 14:54:06,687 INFO [RS:0;jenkins-hbase4:35417-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 68494e94cc440349573fb09884f7f6bc/cf1 in testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:06,687 INFO [RS:0;jenkins-hbase4:35417-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 68494e94cc440349573fb09884f7f6bc/cf2 in testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:06,687 INFO [RS:0;jenkins-hbase4:35417-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp, totalSize=34.9 K 2023-04-22 14:54:06,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:06,687 INFO [RS:0;jenkins-hbase4:35417-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp, totalSize=34.9 K 2023-04-22 14:54:06,688 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:06,688 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] compactions.Compactor(207): Compacting a55a3c6d837b439aac7d739c3b624761, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175238381 2023-04-22 14:54:06,688 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=47 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:06,688 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] compactions.Compactor(207): Compacting 3533885279e04f4a9dfabbab87abf200, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175238381 2023-04-22 14:54:06,688 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175246688"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175246688"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175246688"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175246688"}]},"ts":"1682175246688"} 2023-04-22 14:54:06,689 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] compactions.Compactor(207): Compacting 17172f828c0e4f8d8e82ad29a1c72456, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175238592 2023-04-22 14:54:06,689 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] compactions.Compactor(207): Compacting b12cee7be67d466cbe1457f8bc4723f0, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175238592 2023-04-22 14:54:06,689 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] compactions.Compactor(207): Compacting df2c1a7be9ed4995b2d8b34082133c1c, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175238876 2023-04-22 14:54:06,689 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] compactions.Compactor(207): Compacting ac35848cc24d423e8894a0aea7589d54, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175238876 2023-04-22 14:54:06,689 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] compactions.Compactor(207): Compacting 11c59ba427164756ae1d366488ee2d17, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175239133 2023-04-22 14:54:06,690 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] compactions.Compactor(207): Compacting 86caaf5465a24975808612073073257a, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175239133 2023-04-22 14:54:06,690 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] compactions.Compactor(207): Compacting a30570688f314130ab142b7829fe562f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175239406 2023-04-22 14:54:06,690 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] compactions.Compactor(207): Compacting 9437fc31aa604e9490bd2fdbf9b59292, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175239406 2023-04-22 14:54:06,691 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] compactions.Compactor(207): Compacting 5e946366082547e6b5750eceb6a14f2a, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175239636 2023-04-22 14:54:06,691 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] compactions.Compactor(207): Compacting 31eeb60011a54db1aa76957e8575c767, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175239636 2023-04-22 14:54:06,691 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] compactions.Compactor(207): Compacting f23d308a83544ae3ac5a7790c861929a, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175239869 2023-04-22 14:54:06,691 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] compactions.Compactor(207): Compacting 6d189a279acd4fd3874f0c48349dc663, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175239869 2023-04-22 14:54:06,692 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-22 14:54:06,692 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; OpenRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 in 234 msec 2023-04-22 14:54:06,694 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN in 1.5570 sec 2023-04-22 14:54:06,705 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(335): procId=44 exception: org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested 2023-04-22 14:54:06,708 DEBUG [Listener at localhost/43403] hbase.HBaseTestingUtility(3430): Waiting until all regions of table testRollbackAndDoubleExecution get assigned. Timeout = 1500ms 2023-04-22 14:54:06,710 INFO [Listener at localhost/43403] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-04-22 14:54:06,713 INFO [RS:0;jenkins-hbase4:35417-longCompactions-0] throttle.PressureAwareThroughputController(145): 68494e94cc440349573fb09884f7f6bc#cf2#compaction#50 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:06,713 INFO [RS:0;jenkins-hbase4:35417-shortCompactions-0] throttle.PressureAwareThroughputController(145): 68494e94cc440349573fb09884f7f6bc#cf1#compaction#51 average throughput is 1.72 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:06,714 INFO [Listener at localhost/43403] hbase.HBaseTestingUtility(3484): All regions for table testRollbackAndDoubleExecution assigned to meta. Checking AM states. 2023-04-22 14:54:06,715 INFO [Listener at localhost/43403] hbase.Waiter(180): Waiting up to [1,500] milli-secs(wait.for.ratio=[1]) 2023-04-22 14:54:06,715 INFO [Listener at localhost/43403] hbase.HBaseTestingUtility(3504): All regions for table testRollbackAndDoubleExecution assigned. 2023-04-22 14:54:06,746 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf1/7f4472beb19e40aeae930ba02e9f57fb as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/7f4472beb19e40aeae930ba02e9f57fb 2023-04-22 14:54:06,746 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/.tmp/cf2/b81274602b1e4445bdef68dadcf60e6d as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b81274602b1e4445bdef68dadcf60e6d 2023-04-22 14:54:06,752 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackAndDoubleExecution' 2023-04-22 14:54:06,752 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackAndDoubleExecution' 2023-04-22 14:54:06,755 INFO [RS:0;jenkins-hbase4:35417-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 68494e94cc440349573fb09884f7f6bc/cf1 of 68494e94cc440349573fb09884f7f6bc into 7f4472beb19e40aeae930ba02e9f57fb(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:06,755 INFO [RS:0;jenkins-hbase4:35417-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 68494e94cc440349573fb09884f7f6bc/cf2 of 68494e94cc440349573fb09884f7f6bc into b81274602b1e4445bdef68dadcf60e6d(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:06,755 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:54:06,755 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:54:06,755 INFO [RS:0;jenkins-hbase4:35417-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc., storeName=68494e94cc440349573fb09884f7f6bc/cf1, priority=9, startTime=1682175246684; duration=0sec 2023-04-22 14:54:06,755 INFO [RS:0;jenkins-hbase4:35417-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc., storeName=68494e94cc440349573fb09884f7f6bc/cf2, priority=9, startTime=1682175246684; duration=0sec 2023-04-22 14:54:06,756 DEBUG [RS:0;jenkins-hbase4:35417-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:06,756 DEBUG [RS:0;jenkins-hbase4:35417-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:06,769 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:54:06,771 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testRollbackAndDoubleExecution 2023-04-22 14:54:06,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRollbackAndDoubleExecution 2023-04-22 14:54:06,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:06,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-04-22 14:54:06,776 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175246776"}]},"ts":"1682175246776"} 2023-04-22 14:54:06,777 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLING in hbase:meta 2023-04-22 14:54:06,779 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackAndDoubleExecution to state=DISABLING 2023-04-22 14:54:06,780 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN}] 2023-04-22 14:54:06,781 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN 2023-04-22 14:54:06,781 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:06,781 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175246781"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175246781"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175246781"}]},"ts":"1682175246781"} 2023-04-22 14:54:06,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:54:06,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-04-22 14:54:06,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 68494e94cc440349573fb09884f7f6bc, disabling compactions & flushes 2023-04-22 14:54:06,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:06,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:06,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. after waiting 0 ms 2023-04-22 14:54:06,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:06,936 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663] to archive 2023-04-22 14:54:06,937 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:06,939 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a55a3c6d837b439aac7d739c3b624761 2023-04-22 14:54:06,940 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/17172f828c0e4f8d8e82ad29a1c72456 2023-04-22 14:54:06,941 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/df2c1a7be9ed4995b2d8b34082133c1c 2023-04-22 14:54:06,942 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/11c59ba427164756ae1d366488ee2d17 2023-04-22 14:54:06,943 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/a30570688f314130ab142b7829fe562f 2023-04-22 14:54:06,944 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/31eeb60011a54db1aa76957e8575c767 2023-04-22 14:54:06,945 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/6d189a279acd4fd3874f0c48349dc663 2023-04-22 14:54:06,950 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a] to archive 2023-04-22 14:54:06,951 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:06,952 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/3533885279e04f4a9dfabbab87abf200 2023-04-22 14:54:06,953 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b12cee7be67d466cbe1457f8bc4723f0 2023-04-22 14:54:06,954 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/ac35848cc24d423e8894a0aea7589d54 2023-04-22 14:54:06,955 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/86caaf5465a24975808612073073257a 2023-04-22 14:54:06,956 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/9437fc31aa604e9490bd2fdbf9b59292 2023-04-22 14:54:06,957 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/5e946366082547e6b5750eceb6a14f2a 2023-04-22 14:54:06,958 DEBUG [StoreCloser-testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/f23d308a83544ae3ac5a7790c861929a 2023-04-22 14:54:06,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:54:06,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:54:06,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc. 2023-04-22 14:54:06,963 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 68494e94cc440349573fb09884f7f6bc: 2023-04-22 14:54:06,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:06,965 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=68494e94cc440349573fb09884f7f6bc, regionState=CLOSED 2023-04-22 14:54:06,965 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175246965"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175246965"}]},"ts":"1682175246965"} 2023-04-22 14:54:06,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-22 14:54:06,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; CloseRegionProcedure 68494e94cc440349573fb09884f7f6bc, server=jenkins-hbase4.apache.org,35417,1682175218529 in 184 msec 2023-04-22 14:54:06,970 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=50, resume processing ppid=49 2023-04-22 14:54:06,971 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, UNASSIGN in 189 msec 2023-04-22 14:54:06,971 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175246971"}]},"ts":"1682175246971"} 2023-04-22 14:54:06,973 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackAndDoubleExecution, state=DISABLED in hbase:meta 2023-04-22 14:54:06,974 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackAndDoubleExecution to state=DISABLED 2023-04-22 14:54:06,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution in 203 msec 2023-04-22 14:54:07,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-04-22 14:54:07,079 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackAndDoubleExecution, procId: 49 completed 2023-04-22 14:54:07,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRollbackAndDoubleExecution 2023-04-22 14:54:07,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:07,082 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=52, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:07,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=52 2023-04-22 14:54:07,083 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=52, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:07,086 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:07,088 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/recovered.edits] 2023-04-22 14:54:07,091 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/7f4472beb19e40aeae930ba02e9f57fb to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf1/7f4472beb19e40aeae930ba02e9f57fb 2023-04-22 14:54:07,093 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b81274602b1e4445bdef68dadcf60e6d to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/cf2/b81274602b1e4445bdef68dadcf60e6d 2023-04-22 14:54:07,096 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc/recovered.edits/90.seqid 2023-04-22 14:54:07,097 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackAndDoubleExecution/68494e94cc440349573fb09884f7f6bc 2023-04-22 14:54:07,097 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackAndDoubleExecution regions 2023-04-22 14:54:07,100 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=52, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:07,102 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackAndDoubleExecution from hbase:meta 2023-04-22 14:54:07,103 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackAndDoubleExecution' descriptor. 2023-04-22 14:54:07,105 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=52, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:07,105 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackAndDoubleExecution' from region states. 2023-04-22 14:54:07,105 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175247105"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:07,106 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 14:54:07,106 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 68494e94cc440349573fb09884f7f6bc, NAME => 'testRollbackAndDoubleExecution,,1682175237877.68494e94cc440349573fb09884f7f6bc.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 14:54:07,106 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackAndDoubleExecution' as deleted. 2023-04-22 14:54:07,107 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175247106"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:07,108 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackAndDoubleExecution state from META 2023-04-22 14:54:07,110 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=52, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:54:07,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution in 30 msec 2023-04-22 14:54:07,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=52 2023-04-22 14:54:07,184 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackAndDoubleExecution, procId: 52 completed 2023-04-22 14:54:07,199 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackAndDoubleExecution Thread=398 (was 398), OpenFileDescriptor=860 (was 846) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=490 (was 428) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=12592 (was 12676) 2023-04-22 14:54:07,214 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=398, OpenFileDescriptor=860, MaxFileDescriptor=60000, SystemLoadAverage=490, ProcessCount=172, AvailableMemoryMB=12591 2023-04-22 14:54:07,214 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:54:07,219 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:54:07,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:54:07,221 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:54:07,223 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:54:07,224 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:54:07,226 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,226 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e empty. 2023-04-22 14:54:07,227 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,227 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-04-22 14:54:07,237 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/.tabledesc/.tableinfo.0000000001 2023-04-22 14:54:07,238 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(7675): creating {ENCODED => efad26bc745f89498b20ab3139f9002e, NAME => 'testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionNoStoreFile', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:54:07,248 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:07,249 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1604): Closing efad26bc745f89498b20ab3139f9002e, disabling compactions & flushes 2023-04-22 14:54:07,249 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,249 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,249 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. after waiting 0 ms 2023-04-22 14:54:07,249 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,249 INFO [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,249 DEBUG [RegionOpenAndInit-testSplitTableRegionNoStoreFile-pool-0] regionserver.HRegion(1558): Region close journal for efad26bc745f89498b20ab3139f9002e: 2023-04-22 14:54:07,252 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:54:07,253 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1682175247253"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175247253"}]},"ts":"1682175247253"} 2023-04-22 14:54:07,255 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:54:07,255 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:54:07,256 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175247255"}]},"ts":"1682175247255"} 2023-04-22 14:54:07,257 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLING in hbase:meta 2023-04-22 14:54:07,259 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:54:07,261 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:54:07,261 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:54:07,261 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:54:07,261 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:54:07,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=efad26bc745f89498b20ab3139f9002e, ASSIGN}] 2023-04-22 14:54:07,262 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=efad26bc745f89498b20ab3139f9002e, ASSIGN 2023-04-22 14:54:07,263 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=54, ppid=53, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=efad26bc745f89498b20ab3139f9002e, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:54:07,413 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:54:07,415 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=efad26bc745f89498b20ab3139f9002e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:07,415 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1682175247415"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175247415"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175247415"}]},"ts":"1682175247415"} 2023-04-22 14:54:07,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE; OpenRegionProcedure efad26bc745f89498b20ab3139f9002e, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:54:07,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => efad26bc745f89498b20ab3139f9002e, NAME => 'testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:54:07,576 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:07,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:07,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,577 INFO [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,579 DEBUG [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/cf1 2023-04-22 14:54:07,579 DEBUG [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/cf1 2023-04-22 14:54:07,579 INFO [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region efad26bc745f89498b20ab3139f9002e columnFamilyName cf1 2023-04-22 14:54:07,580 INFO [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] regionserver.HStore(310): Store=efad26bc745f89498b20ab3139f9002e/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:07,580 INFO [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,581 DEBUG [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/cf2 2023-04-22 14:54:07,581 DEBUG [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/cf2 2023-04-22 14:54:07,582 INFO [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region efad26bc745f89498b20ab3139f9002e columnFamilyName cf2 2023-04-22 14:54:07,582 INFO [StoreOpener-efad26bc745f89498b20ab3139f9002e-1] regionserver.HStore(310): Store=efad26bc745f89498b20ab3139f9002e/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:07,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,583 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:54:07,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:54:07,590 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened efad26bc745f89498b20ab3139f9002e; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72496853, jitterRate=0.08028729259967804}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:54:07,590 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for efad26bc745f89498b20ab3139f9002e: 2023-04-22 14:54:07,591 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e., pid=55, masterSystemTime=1682175247568 2023-04-22 14:54:07,592 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,593 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=efad26bc745f89498b20ab3139f9002e, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:07,593 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1682175247593"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175247593"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175247593"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175247593"}]},"ts":"1682175247593"} 2023-04-22 14:54:07,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-22 14:54:07,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; OpenRegionProcedure efad26bc745f89498b20ab3139f9002e, server=jenkins-hbase4.apache.org,36267,1682175218583 in 178 msec 2023-04-22 14:54:07,598 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=54, resume processing ppid=53 2023-04-22 14:54:07,598 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=53, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=efad26bc745f89498b20ab3139f9002e, ASSIGN in 335 msec 2023-04-22 14:54:07,599 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:54:07,599 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175247599"}]},"ts":"1682175247599"} 2023-04-22 14:54:07,600 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=ENABLED in hbase:meta 2023-04-22 14:54:07,602 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=53, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionNoStoreFile execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:54:07,603 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile in 382 msec 2023-04-22 14:54:07,722 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:07,723 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:07,724 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:07,724 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:07,725 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=56, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:07,727 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=efad26bc745f89498b20ab3139f9002e, UNASSIGN}] 2023-04-22 14:54:07,727 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=57, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=efad26bc745f89498b20ab3139f9002e, UNASSIGN 2023-04-22 14:54:07,728 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=efad26bc745f89498b20ab3139f9002e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:07,728 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1682175247728"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175247728"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175247728"}]},"ts":"1682175247728"} 2023-04-22 14:54:07,729 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE; CloseRegionProcedure efad26bc745f89498b20ab3139f9002e, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:54:07,881 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing efad26bc745f89498b20ab3139f9002e, disabling compactions & flushes 2023-04-22 14:54:07,882 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. after waiting 0 ms 2023-04-22 14:54:07,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,886 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-22 14:54:07,887 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:54:07,887 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e. 2023-04-22 14:54:07,887 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for efad26bc745f89498b20ab3139f9002e: 2023-04-22 14:54:07,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:07,889 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=57 updating hbase:meta row=efad26bc745f89498b20ab3139f9002e, regionState=CLOSED 2023-04-22 14:54:07,889 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1682175247889"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175247889"}]},"ts":"1682175247889"} 2023-04-22 14:54:07,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-22 14:54:07,892 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; CloseRegionProcedure efad26bc745f89498b20ab3139f9002e, server=jenkins-hbase4.apache.org,36267,1682175218583 in 161 msec 2023-04-22 14:54:07,893 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=56 2023-04-22 14:54:07,893 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=efad26bc745f89498b20ab3139f9002e, UNASSIGN in 165 msec 2023-04-22 14:54:07,919 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-04-22 14:54:07,921 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=-1 2023-04-22 14:54:07,923 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.","families":{"info":[{"qualifier":"regioninfo","vlen":65,"tag":[],"timestamp":"1682175247922"},{"qualifier":"splitA","vlen":67,"tag":[],"timestamp":"1682175247922"},{"qualifier":"splitB","vlen":67,"tag":[],"timestamp":"1682175247922"}]},"ts":"1682175247922"} 2023-04-22 14:54:07,923 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175247922"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175247922"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175247922"}]},"ts":"1682175247922"} 2023-04-22 14:54:07,923 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175247922"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175247922"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175247922"}]},"ts":"1682175247922"} 2023-04-22 14:54:07,926 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=3773e0571cc154fea12efd9885f8a72e, ASSIGN}, {pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=4d87b2e1a535264e1cee0f8f8d63d62e, ASSIGN}] 2023-04-22 14:54:07,927 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=4d87b2e1a535264e1cee0f8f8d63d62e, ASSIGN 2023-04-22 14:54:07,927 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=60, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=4d87b2e1a535264e1cee0f8f8d63d62e, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:54:07,928 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=3773e0571cc154fea12efd9885f8a72e, ASSIGN 2023-04-22 14:54:07,928 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=56, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=3773e0571cc154fea12efd9885f8a72e, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:54:08,077 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-22 14:54:08,079 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=4d87b2e1a535264e1cee0f8f8d63d62e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:08,079 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175248079"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175248079"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175248079"}]},"ts":"1682175248079"} 2023-04-22 14:54:08,081 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=60, state=RUNNABLE; OpenRegionProcedure 4d87b2e1a535264e1cee0f8f8d63d62e, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:54:08,083 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3773e0571cc154fea12efd9885f8a72e, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:08,083 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175248083"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175248083"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175248083"}]},"ts":"1682175248083"} 2023-04-22 14:54:08,085 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=62, ppid=59, state=RUNNABLE; OpenRegionProcedure 3773e0571cc154fea12efd9885f8a72e, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:54:08,235 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e. 2023-04-22 14:54:08,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 4d87b2e1a535264e1cee0f8f8d63d62e, NAME => 'testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e.', STARTKEY => '41', ENDKEY => ''} 2023-04-22 14:54:08,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:08,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:08,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,238 INFO [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,239 DEBUG [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/cf1 2023-04-22 14:54:08,239 DEBUG [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/cf1 2023-04-22 14:54:08,240 INFO [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4d87b2e1a535264e1cee0f8f8d63d62e columnFamilyName cf1 2023-04-22 14:54:08,240 INFO [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] regionserver.HStore(310): Store=4d87b2e1a535264e1cee0f8f8d63d62e/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:08,240 INFO [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,242 DEBUG [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/cf2 2023-04-22 14:54:08,242 DEBUG [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/cf2 2023-04-22 14:54:08,242 INFO [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 4d87b2e1a535264e1cee0f8f8d63d62e columnFamilyName cf2 2023-04-22 14:54:08,243 INFO [StoreOpener-4d87b2e1a535264e1cee0f8f8d63d62e-1] regionserver.HStore(310): Store=4d87b2e1a535264e1cee0f8f8d63d62e/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:08,243 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:54:08,247 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 4d87b2e1a535264e1cee0f8f8d63d62e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73920353, jitterRate=0.10149909555912018}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:54:08,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 4d87b2e1a535264e1cee0f8f8d63d62e: 2023-04-22 14:54:08,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e., pid=61, masterSystemTime=1682175248232 2023-04-22 14:54:08,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e. 2023-04-22 14:54:08,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e. 2023-04-22 14:54:08,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e. 2023-04-22 14:54:08,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3773e0571cc154fea12efd9885f8a72e, NAME => 'testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e.', STARTKEY => '', ENDKEY => '41'} 2023-04-22 14:54:08,251 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=60 updating hbase:meta row=4d87b2e1a535264e1cee0f8f8d63d62e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:08,251 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175248251"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175248251"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175248251"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175248251"}]},"ts":"1682175248251"} 2023-04-22 14:54:08,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:08,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionNoStoreFile 3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:08,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,252 INFO [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=60 2023-04-22 14:54:08,254 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=60, state=SUCCESS; OpenRegionProcedure 4d87b2e1a535264e1cee0f8f8d63d62e, server=jenkins-hbase4.apache.org,36267,1682175218583 in 171 msec 2023-04-22 14:54:08,254 DEBUG [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/cf1 2023-04-22 14:54:08,254 DEBUG [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/cf1 2023-04-22 14:54:08,255 INFO [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3773e0571cc154fea12efd9885f8a72e columnFamilyName cf1 2023-04-22 14:54:08,255 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=4d87b2e1a535264e1cee0f8f8d63d62e, ASSIGN in 328 msec 2023-04-22 14:54:08,255 INFO [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] regionserver.HStore(310): Store=3773e0571cc154fea12efd9885f8a72e/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:08,255 INFO [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,257 DEBUG [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/cf2 2023-04-22 14:54:08,257 DEBUG [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/cf2 2023-04-22 14:54:08,257 INFO [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3773e0571cc154fea12efd9885f8a72e columnFamilyName cf2 2023-04-22 14:54:08,258 INFO [StoreOpener-3773e0571cc154fea12efd9885f8a72e-1] regionserver.HStore(310): Store=3773e0571cc154fea12efd9885f8a72e/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:08,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionNoStoreFile descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:54:08,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3773e0571cc154fea12efd9885f8a72e; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63788811, jitterRate=-0.04947264492511749}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:54:08,263 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3773e0571cc154fea12efd9885f8a72e: 2023-04-22 14:54:08,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e., pid=62, masterSystemTime=1682175248232 2023-04-22 14:54:08,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e. 2023-04-22 14:54:08,265 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e. 2023-04-22 14:54:08,265 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=3773e0571cc154fea12efd9885f8a72e, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:08,266 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175248265"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175248265"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175248265"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175248265"}]},"ts":"1682175248265"} 2023-04-22 14:54:08,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=62, resume processing ppid=59 2023-04-22 14:54:08,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, ppid=59, state=SUCCESS; OpenRegionProcedure 3773e0571cc154fea12efd9885f8a72e, server=jenkins-hbase4.apache.org,36267,1682175218583 in 182 msec 2023-04-22 14:54:08,270 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=56 2023-04-22 14:54:08,270 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=56, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=3773e0571cc154fea12efd9885f8a72e, ASSIGN in 342 msec 2023-04-22 14:54:08,271 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e in 548 msec 2023-04-22 14:54:08,478 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-22 14:54:08,480 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:54:08,482 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-04-22 14:54:08,486 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175248485"}]},"ts":"1682175248485"} 2023-04-22 14:54:08,487 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLING in hbase:meta 2023-04-22 14:54:08,488 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionNoStoreFile to state=DISABLING 2023-04-22 14:54:08,489 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=3773e0571cc154fea12efd9885f8a72e, UNASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=4d87b2e1a535264e1cee0f8f8d63d62e, UNASSIGN}] 2023-04-22 14:54:08,490 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=4d87b2e1a535264e1cee0f8f8d63d62e, UNASSIGN 2023-04-22 14:54:08,490 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=4d87b2e1a535264e1cee0f8f8d63d62e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:08,490 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175248490"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175248490"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175248490"}]},"ts":"1682175248490"} 2023-04-22 14:54:08,492 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=66, ppid=65, state=RUNNABLE; CloseRegionProcedure 4d87b2e1a535264e1cee0f8f8d63d62e, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:54:08,494 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=3773e0571cc154fea12efd9885f8a72e, UNASSIGN 2023-04-22 14:54:08,494 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=3773e0571cc154fea12efd9885f8a72e, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:08,494 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175248494"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175248494"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175248494"}]},"ts":"1682175248494"} 2023-04-22 14:54:08,496 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=67, ppid=64, state=RUNNABLE; CloseRegionProcedure 3773e0571cc154fea12efd9885f8a72e, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:54:08,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-04-22 14:54:08,644 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,645 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 4d87b2e1a535264e1cee0f8f8d63d62e, disabling compactions & flushes 2023-04-22 14:54:08,645 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e. 2023-04-22 14:54:08,645 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e. 2023-04-22 14:54:08,645 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e. after waiting 0 ms 2023-04-22 14:54:08,645 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e. 2023-04-22 14:54:08,649 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-04-22 14:54:08,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:54:08,650 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e. 2023-04-22 14:54:08,650 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 4d87b2e1a535264e1cee0f8f8d63d62e: 2023-04-22 14:54:08,651 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,652 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3773e0571cc154fea12efd9885f8a72e, disabling compactions & flushes 2023-04-22 14:54:08,653 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e. 2023-04-22 14:54:08,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e. 2023-04-22 14:54:08,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e. after waiting 0 ms 2023-04-22 14:54:08,653 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e. 2023-04-22 14:54:08,653 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=4d87b2e1a535264e1cee0f8f8d63d62e, regionState=CLOSED 2023-04-22 14:54:08,653 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175248653"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175248653"}]},"ts":"1682175248653"} 2023-04-22 14:54:08,657 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-04-22 14:54:08,657 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=66, resume processing ppid=65 2023-04-22 14:54:08,657 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=65, state=SUCCESS; CloseRegionProcedure 4d87b2e1a535264e1cee0f8f8d63d62e, server=jenkins-hbase4.apache.org,36267,1682175218583 in 164 msec 2023-04-22 14:54:08,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:54:08,658 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e. 2023-04-22 14:54:08,658 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3773e0571cc154fea12efd9885f8a72e: 2023-04-22 14:54:08,659 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=4d87b2e1a535264e1cee0f8f8d63d62e, UNASSIGN in 169 msec 2023-04-22 14:54:08,659 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,660 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=3773e0571cc154fea12efd9885f8a72e, regionState=CLOSED 2023-04-22 14:54:08,660 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175248660"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175248660"}]},"ts":"1682175248660"} 2023-04-22 14:54:08,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=67, resume processing ppid=64 2023-04-22 14:54:08,663 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=64, state=SUCCESS; CloseRegionProcedure 3773e0571cc154fea12efd9885f8a72e, server=jenkins-hbase4.apache.org,36267,1682175218583 in 165 msec 2023-04-22 14:54:08,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-22 14:54:08,664 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionNoStoreFile, region=3773e0571cc154fea12efd9885f8a72e, UNASSIGN in 174 msec 2023-04-22 14:54:08,665 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175248665"}]},"ts":"1682175248665"} 2023-04-22 14:54:08,666 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionNoStoreFile, state=DISABLED in hbase:meta 2023-04-22 14:54:08,669 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionNoStoreFile to state=DISABLED 2023-04-22 14:54:08,671 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile in 188 msec 2023-04-22 14:54:08,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-04-22 14:54:08,788 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 63 completed 2023-04-22 14:54:08,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,791 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=68, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,792 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=68, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=68 2023-04-22 14:54:08,796 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:08,796 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,796 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,798 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/recovered.edits] 2023-04-22 14:54:08,798 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/recovered.edits] 2023-04-22 14:54:08,798 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/recovered.edits] 2023-04-22 14:54:08,804 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/recovered.edits/7.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e/recovered.edits/7.seqid 2023-04-22 14:54:08,804 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/recovered.edits/7.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e/recovered.edits/7.seqid 2023-04-22 14:54:08,804 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/recovered.edits/4.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e/recovered.edits/4.seqid 2023-04-22 14:54:08,805 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:54:08,805 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/3773e0571cc154fea12efd9885f8a72e 2023-04-22 14:54:08,805 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionNoStoreFile/efad26bc745f89498b20ab3139f9002e 2023-04-22 14:54:08,805 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionNoStoreFile regions 2023-04-22 14:54:08,807 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=68, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,809 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionNoStoreFile from hbase:meta 2023-04-22 14:54:08,810 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionNoStoreFile' descriptor. 2023-04-22 14:54:08,811 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=68, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,811 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionNoStoreFile' from region states. 2023-04-22 14:54:08,812 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175248811"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:08,812 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175248811"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:08,812 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175248811"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:08,813 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-22 14:54:08,813 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => efad26bc745f89498b20ab3139f9002e, NAME => 'testSplitTableRegionNoStoreFile,,1682175247220.efad26bc745f89498b20ab3139f9002e.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 3773e0571cc154fea12efd9885f8a72e, NAME => 'testSplitTableRegionNoStoreFile,,1682175247722.3773e0571cc154fea12efd9885f8a72e.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => 4d87b2e1a535264e1cee0f8f8d63d62e, NAME => 'testSplitTableRegionNoStoreFile,41,1682175247722.4d87b2e1a535264e1cee0f8f8d63d62e.', STARTKEY => '41', ENDKEY => ''}] 2023-04-22 14:54:08,813 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionNoStoreFile' as deleted. 2023-04-22 14:54:08,813 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionNoStoreFile","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175248813"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:08,815 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionNoStoreFile state from META 2023-04-22 14:54:08,816 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=68, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:54:08,817 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile in 27 msec 2023-04-22 14:54:08,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=68 2023-04-22 14:54:08,893 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionNoStoreFile, procId: 68 completed 2023-04-22 14:54:08,908 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionNoStoreFile Thread=402 (was 398) Potentially hanging thread: hconnection-0x6454e058-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-261843219_15 at /127.0.0.1:58476 [Waiting for operation #21] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-261843219_15 at /127.0.0.1:40048 [Waiting for operation #41] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=860 (was 860), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=450 (was 490), ProcessCount=172 (was 172), AvailableMemoryMB=12596 (was 12591) - AvailableMemoryMB LEAK? - 2023-04-22 14:54:08,923 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=402, OpenFileDescriptor=860, MaxFileDescriptor=60000, SystemLoadAverage=450, ProcessCount=172, AvailableMemoryMB=12595 2023-04-22 14:54:08,923 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:54:08,925 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:54:08,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:54:08,927 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:08,928 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:54:08,929 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:54:08,931 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:08,931 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824 empty. 2023-04-22 14:54:08,932 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:08,932 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-04-22 14:54:08,947 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/.tabledesc/.tableinfo.0000000001 2023-04-22 14:54:08,948 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(7675): creating {ENCODED => 001cf566a5015e8ef1438c27e2e9c824, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionDeletedRowsDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:54:08,957 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:08,957 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1604): Closing 001cf566a5015e8ef1438c27e2e9c824, disabling compactions & flushes 2023-04-22 14:54:08,957 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:08,957 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:08,957 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. after waiting 0 ms 2023-04-22 14:54:08,957 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:08,957 INFO [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:08,957 DEBUG [RegionOpenAndInit-testSplitTableRegionDeletedRowsDaughter-pool-0] regionserver.HRegion(1558): Region close journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:08,959 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:54:08,960 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1682175248960"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175248960"}]},"ts":"1682175248960"} 2023-04-22 14:54:08,962 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:54:08,962 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:54:08,963 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175248962"}]},"ts":"1682175248962"} 2023-04-22 14:54:08,964 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLING in hbase:meta 2023-04-22 14:54:08,968 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:54:08,969 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:54:08,969 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:54:08,969 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:54:08,969 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:54:08,969 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=001cf566a5015e8ef1438c27e2e9c824, ASSIGN}] 2023-04-22 14:54:08,971 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=001cf566a5015e8ef1438c27e2e9c824, ASSIGN 2023-04-22 14:54:08,971 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=70, ppid=69, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=001cf566a5015e8ef1438c27e2e9c824, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33597,1682175218631; forceNewPlan=false, retain=false 2023-04-22 14:54:09,122 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:54:09,123 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=001cf566a5015e8ef1438c27e2e9c824, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,123 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1682175249123"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175249123"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175249123"}]},"ts":"1682175249123"} 2023-04-22 14:54:09,125 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=70, state=RUNNABLE; OpenRegionProcedure 001cf566a5015e8ef1438c27e2e9c824, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:54:09,278 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,278 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-22 14:54:09,280 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39424, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-22 14:54:09,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:09,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 001cf566a5015e8ef1438c27e2e9c824, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:54:09,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:09,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:09,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:09,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:09,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:09,291 INFO [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:09,293 DEBUG [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1 2023-04-22 14:54:09,293 DEBUG [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1 2023-04-22 14:54:09,294 INFO [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 001cf566a5015e8ef1438c27e2e9c824 columnFamilyName cf1 2023-04-22 14:54:09,294 INFO [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] regionserver.HStore(310): Store=001cf566a5015e8ef1438c27e2e9c824/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:09,294 INFO [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:09,296 DEBUG [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2 2023-04-22 14:54:09,296 DEBUG [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2 2023-04-22 14:54:09,296 INFO [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 001cf566a5015e8ef1438c27e2e9c824 columnFamilyName cf2 2023-04-22 14:54:09,297 INFO [StoreOpener-001cf566a5015e8ef1438c27e2e9c824-1] regionserver.HStore(310): Store=001cf566a5015e8ef1438c27e2e9c824/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:09,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:09,298 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:09,300 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:54:09,302 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:09,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:54:09,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 001cf566a5015e8ef1438c27e2e9c824; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74329893, jitterRate=0.10760171711444855}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:54:09,305 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:09,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824., pid=71, masterSystemTime=1682175249277 2023-04-22 14:54:09,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:09,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:09,311 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=70 updating hbase:meta row=001cf566a5015e8ef1438c27e2e9c824, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,311 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1682175249311"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175249311"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175249311"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175249311"}]},"ts":"1682175249311"} 2023-04-22 14:54:09,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=70 2023-04-22 14:54:09,314 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=70, state=SUCCESS; OpenRegionProcedure 001cf566a5015e8ef1438c27e2e9c824, server=jenkins-hbase4.apache.org,33597,1682175218631 in 187 msec 2023-04-22 14:54:09,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=69 2023-04-22 14:54:09,315 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=69, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=001cf566a5015e8ef1438c27e2e9c824, ASSIGN in 345 msec 2023-04-22 14:54:09,316 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:54:09,316 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175249316"}]},"ts":"1682175249316"} 2023-04-22 14:54:09,317 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=ENABLED in hbase:meta 2023-04-22 14:54:09,319 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=69, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:54:09,320 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 393 msec 2023-04-22 14:54:09,430 DEBUG [Listener at localhost/43403] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-22 14:54:09,435 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39428, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-22 14:54:09,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:09,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,443 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:09,443 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,443 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-22 14:54:09,443 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-22 14:54:09,444 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,444 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,445 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,445 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,445 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,445 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,446 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,446 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,447 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,448 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:09,448 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,448 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:09,448 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:09,448 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:09,449 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:09,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:09,450 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:09,450 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:09,450 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. started... 2023-04-22 14:54:09,452 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-22 14:54:09,454 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,454 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,454 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:09,463 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/8b161201a2364c3eba9b9cf63655c553 2023-04-22 14:54:09,480 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/24bf657f679f4277a86b89e4687e82de 2023-04-22 14:54:09,485 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/8b161201a2364c3eba9b9cf63655c553 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553 2023-04-22 14:54:09,491 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:54:09,492 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/24bf657f679f4277a86b89e4687e82de as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de 2023-04-22 14:54:09,496 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:54:09,498 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 46ms, sequenceid=6, compaction requested=false 2023-04-22 14:54:09,498 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:09,498 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:09,498 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:09,498 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:09,499 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:09,500 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,500 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:09,501 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,501 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:09,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:09,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-22 14:54:09,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1df54b5a[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:09,503 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-22 14:54:09,503 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,504 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,504 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,504 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,504 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:09,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,504 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:09,504 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:54:09,504 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,505 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:09,504 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:09,505 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,505 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:54:09,505 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:09,505 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:09,505 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:54:09,506 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:09,506 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,506 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:09,506 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:09,507 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:09,507 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,507 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:09,508 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:09,508 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,508 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:09,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:54:09,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:54:09,511 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-22 14:54:09,511 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:09,511 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:09,511 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,511 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:09,512 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:09,513 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,513 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,514 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,514 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:09,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:09,516 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,516 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,516 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:09,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:09,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,528 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,529 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,528 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,529 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:09,529 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,529 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,529 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,529 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,529 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,629 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:09,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:09,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:09,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:09,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,646 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:09,646 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,646 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-22 14:54:09,646 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-22 14:54:09,647 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,647 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,648 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,648 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,648 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,648 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,648 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,648 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:09,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:09,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:09,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:09,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:09,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:09,650 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:09,650 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. started... 2023-04-22 14:54:09,650 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:54:09,652 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,653 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,653 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,653 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:09,653 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,653 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:09,665 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e 2023-04-22 14:54:09,680 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/39955a28897d41b7af911cc88a9a00f3 2023-04-22 14:54:09,684 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e 2023-04-22 14:54:09,687 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:54:09,688 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/39955a28897d41b7af911cc88a9a00f3 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3 2023-04-22 14:54:09,692 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:54:09,693 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 43ms, sequenceid=19, compaction requested=false 2023-04-22 14:54:09,693 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:09,693 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:09,693 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:09,693 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:09,693 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,693 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:09,693 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:09,695 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:09,695 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:09,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-22 14:54:09,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@325932ee[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:09,697 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-22 14:54:09,697 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,698 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,722 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,722 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,723 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:54:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:54:09,723 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,729 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,729 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:09,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:54:09,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:54:09,732 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-22 14:54:09,733 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:09,733 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:09,733 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,733 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:09,734 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,734 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,734 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,734 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,734 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:09,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,734 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,734 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,735 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:09,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,735 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:09,736 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,738 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:09,738 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,738 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,741 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,741 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,741 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,741 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,741 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,741 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:09,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:09,741 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,742 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,742 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,742 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:09,742 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,742 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:09,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,742 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,743 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,842 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:09,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:09,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:09,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:09,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,857 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:09,857 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,857 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-22 14:54:09,857 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-22 14:54:09,857 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,857 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,860 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,860 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,860 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,860 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,860 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,860 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:09,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:09,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:09,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:09,862 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:09,862 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. started... 2023-04-22 14:54:09,863 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:54:09,863 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,865 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,865 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,865 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:09,865 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,865 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:09,871 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/c5eb2af66eca4eee9ea2de5e97ad9312 2023-04-22 14:54:09,884 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/6ffc2df596cf4051a3d18ba09abfbdc7 2023-04-22 14:54:09,888 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/c5eb2af66eca4eee9ea2de5e97ad9312 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312 2023-04-22 14:54:09,892 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:54:09,893 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/6ffc2df596cf4051a3d18ba09abfbdc7 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7 2023-04-22 14:54:09,897 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:54:09,897 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 34ms, sequenceid=32, compaction requested=true 2023-04-22 14:54:09,898 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:09,898 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:09,898 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:09,898 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:09,898 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,898 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:09,898 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:09,903 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,903 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,904 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:09,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-22 14:54:09,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f670738[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:09,905 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-22 14:54:09,905 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:09,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:09,907 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,907 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:09,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:54:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:54:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:09,907 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:54:09,908 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:09,908 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,908 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:09,908 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:09,908 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:09,909 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,909 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:09,914 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:09,914 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:09,914 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:54:09,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:54:09,916 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-22 14:54:09,916 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:09,916 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:09,916 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,916 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:09,918 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:09,918 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,918 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,918 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,919 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,919 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,919 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,921 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,921 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,922 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:09,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,924 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:09,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,925 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:09,925 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:09,924 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:09,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:09,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:09,925 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,925 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:09,925 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,925 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:09,925 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:09,925 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,925 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:09,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,025 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:10,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:10,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:10,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:10,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,040 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,040 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,040 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-22 14:54:10,040 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-22 14:54:10,041 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,041 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,042 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,042 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,042 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,042 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,042 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,042 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,044 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:10,045 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:10,045 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. started... 2023-04-22 14:54:10,045 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,045 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:54:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,047 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,048 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,048 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,054 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/5d0e1730080c447fa7aad847a9b73b10 2023-04-22 14:54:10,065 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/ff0bfaf3fe824d1ca8a37799277b9aba 2023-04-22 14:54:10,068 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/5d0e1730080c447fa7aad847a9b73b10 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5d0e1730080c447fa7aad847a9b73b10 2023-04-22 14:54:10,072 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5d0e1730080c447fa7aad847a9b73b10, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:54:10,073 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/ff0bfaf3fe824d1ca8a37799277b9aba as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/ff0bfaf3fe824d1ca8a37799277b9aba 2023-04-22 14:54:10,076 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/ff0bfaf3fe824d1ca8a37799277b9aba, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:54:10,077 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 32ms, sequenceid=45, compaction requested=true 2023-04-22 14:54:10,077 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:10,077 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,077 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:10,077 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:10,078 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,078 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,078 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,079 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,079 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,080 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,080 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-22 14:54:10,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56ab5ace[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:10,081 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-22 14:54:10,081 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:54:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:54:10,083 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,083 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,083 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:54:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,085 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,086 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,086 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,086 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,086 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,086 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,086 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,086 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:54:10,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:54:10,089 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-22 14:54:10,089 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:10,089 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:10,089 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,089 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:10,096 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,096 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,096 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,096 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,096 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,096 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,096 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,097 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,098 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,098 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,099 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,100 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,100 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,102 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,103 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,103 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:10,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,103 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,102 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,103 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,103 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,103 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,103 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,103 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,103 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,203 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:10,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:10,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:10,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:10,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,218 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,218 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,218 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-22 14:54:10,218 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-22 14:54:10,219 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,219 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,220 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,220 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,220 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,220 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,220 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,220 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,222 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,223 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:10,223 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:10,223 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,223 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. started... 2023-04-22 14:54:10,223 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:54:10,225 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,226 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,243 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/c1ed0410d24a40f4ae9c7db7c0fe1893 2023-04-22 14:54:10,258 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/e6d6d24cf4e343e5b9e33a8b6f8e3b62 2023-04-22 14:54:10,263 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/c1ed0410d24a40f4ae9c7db7c0fe1893 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c1ed0410d24a40f4ae9c7db7c0fe1893 2023-04-22 14:54:10,267 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c1ed0410d24a40f4ae9c7db7c0fe1893, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:54:10,267 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/e6d6d24cf4e343e5b9e33a8b6f8e3b62 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e6d6d24cf4e343e5b9e33a8b6f8e3b62 2023-04-22 14:54:10,271 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e6d6d24cf4e343e5b9e33a8b6f8e3b62, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:54:10,272 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 49ms, sequenceid=58, compaction requested=true 2023-04-22 14:54:10,272 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:10,272 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,273 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:10,273 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:10,273 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,273 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,273 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,274 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,274 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,275 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,275 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-22 14:54:10,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b364da[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:10,277 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-22 14:54:10,277 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,279 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,279 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:10,279 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:54:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,279 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,280 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,280 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,280 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,280 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:54:10,280 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:54:10,281 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,281 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,281 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,282 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,283 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,283 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,283 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,283 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,283 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,283 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:54:10,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:54:10,286 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-22 14:54:10,286 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:10,286 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:10,286 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,286 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:10,288 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,289 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,289 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,290 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,291 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,292 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,292 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,296 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,296 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,296 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,296 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,296 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,296 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:10,296 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,297 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,296 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,296 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,297 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,297 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,297 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:10,297 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,298 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,298 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,298 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,298 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,298 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,298 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,398 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:10,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:10,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:10,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:10,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,413 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,413 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,413 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-22 14:54:10,413 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-22 14:54:10,413 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,413 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,417 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,417 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,417 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,417 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,418 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,418 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,419 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,420 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,420 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,420 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:10,420 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:10,420 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,420 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. started... 2023-04-22 14:54:10,420 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:54:10,421 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,422 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,422 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,422 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,422 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,422 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,429 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/89df11190dda482aaeff39606d5c1c39 2023-04-22 14:54:10,441 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/255122a7d4ac44e191a32667e4e2c68b 2023-04-22 14:54:10,446 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/89df11190dda482aaeff39606d5c1c39 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/89df11190dda482aaeff39606d5c1c39 2023-04-22 14:54:10,449 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/89df11190dda482aaeff39606d5c1c39, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:54:10,450 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/255122a7d4ac44e191a32667e4e2c68b as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/255122a7d4ac44e191a32667e4e2c68b 2023-04-22 14:54:10,454 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/255122a7d4ac44e191a32667e4e2c68b, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:54:10,455 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 35ms, sequenceid=71, compaction requested=true 2023-04-22 14:54:10,455 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:10,455 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,455 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,457 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,457 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,457 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,457 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-22 14:54:10,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a06f440[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:10,459 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-22 14:54:10,459 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,461 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,461 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:54:10,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,461 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:54:10,462 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:54:10,463 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,463 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,463 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,464 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,464 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,464 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,464 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,464 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,464 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,464 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:54:10,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:54:10,468 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-22 14:54:10,468 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:10,468 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:10,468 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,468 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:10,470 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,470 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,470 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,470 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,470 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,470 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,470 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,471 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,472 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,472 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,474 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,474 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,475 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,479 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,480 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,479 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,480 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,480 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:10,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,480 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,480 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:10,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,480 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:10,480 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,481 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,581 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:10,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:10,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:10,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:10,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,595 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,595 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,596 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-22 14:54:10,596 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-22 14:54:10,596 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,596 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,597 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,597 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,597 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,597 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,597 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,598 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:10,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:10,599 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:10,599 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,600 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. started... 2023-04-22 14:54:10,600 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=530 B heapSize=2.33 KB 2023-04-22 14:54:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,601 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,602 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,602 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,602 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,610 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/b8c54a96354d4677a10eece13e6be12c 2023-04-22 14:54:10,613 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b8c54a96354d4677a10eece13e6be12c 2023-04-22 14:54:10,622 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=265 B at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/e29f83f11c3248a28793be2b209de5ef 2023-04-22 14:54:10,625 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e29f83f11c3248a28793be2b209de5ef 2023-04-22 14:54:10,626 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/b8c54a96354d4677a10eece13e6be12c as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c 2023-04-22 14:54:10,629 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b8c54a96354d4677a10eece13e6be12c 2023-04-22 14:54:10,629 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c, entries=9, sequenceid=83, filesize=5.1 K 2023-04-22 14:54:10,630 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/e29f83f11c3248a28793be2b209de5ef as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef 2023-04-22 14:54:10,634 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e29f83f11c3248a28793be2b209de5ef 2023-04-22 14:54:10,634 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef, entries=9, sequenceid=83, filesize=5.1 K 2023-04-22 14:54:10,635 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~530 B/530, heapSize ~2.30 KB/2352, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 35ms, sequenceid=83, compaction requested=true 2023-04-22 14:54:10,635 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:10,635 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,635 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:10,635 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:10,635 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,635 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,635 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,637 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,637 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,638 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,638 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-22 14:54:10,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d8f7eff[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:10,639 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-22 14:54:10,639 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,640 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,640 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,640 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,641 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,641 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,641 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,641 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,641 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,641 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:54:10,641 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:54:10,641 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,641 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,641 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:54:10,641 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:10,642 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,642 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,642 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,647 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,647 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,647 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,647 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,647 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,647 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,647 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:54:10,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:54:10,650 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-22 14:54:10,650 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:10,650 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:10,650 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,650 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:10,652 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,652 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,652 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,652 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,653 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,653 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,654 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,655 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,655 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,657 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:10,658 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,658 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:10,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,658 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:10,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,658 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,658 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,658 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,658 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,658 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,659 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,659 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,659 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,758 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:10,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:10,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:10,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:10,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,769 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,769 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,769 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-22 14:54:10,769 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-22 14:54:10,769 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,769 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,770 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,770 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,770 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,770 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,770 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,770 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:10,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:10,772 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,772 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,773 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. started... 2023-04-22 14:54:10,773 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,773 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-04-22 14:54:10,774 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,775 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,775 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,775 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,775 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,775 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,781 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:10,785 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:10,794 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:10,797 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:10,797 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/368ef33579cc4830990add57e6c07fba as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:10,801 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:10,801 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/368ef33579cc4830990add57e6c07fba, entries=5, sequenceid=91, filesize=5.0 K 2023-04-22 14:54:10,802 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/06f106ab43e844379d17015d273cf8e9 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:10,806 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:10,806 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/06f106ab43e844379d17015d273cf8e9, entries=5, sequenceid=91, filesize=5.0 K 2023-04-22 14:54:10,807 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 34ms, sequenceid=91, compaction requested=true 2023-04-22 14:54:10,807 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:10,807 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,807 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:10,807 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:10,807 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,807 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,807 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,808 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,808 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,809 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,809 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-22 14:54:10,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41fe20aa[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:10,810 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-22 14:54:10,810 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,811 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,811 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,811 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,812 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,812 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,812 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,812 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,812 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,812 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:54:10,812 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,812 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,812 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:54:10,812 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,812 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:54:10,818 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,818 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,818 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,818 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,818 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,819 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,818 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,819 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,819 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,819 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:54:10,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:54:10,821 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-22 14:54:10,821 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:10,821 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:10,821 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,822 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:10,823 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,823 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,823 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,824 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,824 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,826 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,826 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,826 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,834 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,835 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,834 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,835 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,835 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,835 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,835 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:10,835 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,835 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,835 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:10,835 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,935 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:10,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:10,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:54:10,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionDeletedRowsDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:54:10,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,945 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:10,945 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,945 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' 2023-04-22 14:54:10,945 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionDeletedRowsDaughter', kicking off acquire phase on members. 2023-04-22 14:54:10,946 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,946 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,948 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,948 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,948 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,948 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:10,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,949 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,949 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:54:10,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:54:10,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,950 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionDeletedRowsDaughter is still present, removing 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,951 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionDeletedRowsDaughter' with timeout 60000ms 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:54:10,951 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:54:10,951 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. started... 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' starting 'acquire' stage 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,951 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,951 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=250 B heapSize=1.52 KB 2023-04-22 14:54:10,952 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,953 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,953 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,954 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,954 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,954 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,960 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:10,964 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:10,972 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=125 B at sequenceid=99 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:10,975 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:10,976 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/5b4a88346b974583baa74fea26cee27d as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:10,979 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:10,979 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5b4a88346b974583baa74fea26cee27d, entries=5, sequenceid=99, filesize=5.0 K 2023-04-22 14:54:10,980 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/800f49944858458aba0cbdf9ea0ab3cb as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:10,983 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:10,983 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/800f49944858458aba0cbdf9ea0ab3cb, entries=5, sequenceid=99, filesize=5.0 K 2023-04-22 14:54:10,984 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~250 B/250, heapSize ~1.48 KB/1520, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 33ms, sequenceid=99, compaction requested=true 2023-04-22 14:54:10,984 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:10,984 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:10,984 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:54:10,984 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:54:10,984 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,984 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally acquired 2023-04-22 14:54:10,984 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionDeletedRowsDaughter) in zk 2023-04-22 14:54:10,986 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,986 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,986 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,986 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:54:10,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionDeletedRowsDaughter' on coordinator 2023-04-22 14:54:10,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b1109fb[Count = 0] remaining members to acquire global barrier 2023-04-22 14:54:10,988 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionDeletedRowsDaughter' starting 'in-barrier' execution. 2023-04-22 14:54:10,988 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:54:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' received 'reached' from coordinator. 2023-04-22 14:54:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:54:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' locally completed 2023-04-22 14:54:10,990 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionDeletedRowsDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:54:10,990 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,990 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:54:10,991 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,991 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,991 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' has notified controller of completion 2023-04-22 14:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionDeletedRowsDaughter' completed. 2023-04-22 14:54:10,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionDeletedRowsDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:54:10,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionDeletedRowsDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:54:10,995 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionDeletedRowsDaughter' execution completed 2023-04-22 14:54:10,995 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:54:10,995 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:54:10,995 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,995 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionDeletedRowsDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:54:10,997 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:54:10,997 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,997 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,997 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:54:10,997 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,998 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:54:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,999 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:54:10,999 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:10,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:11,000 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:11,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:11,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:11,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:11,003 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:11,003 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:11,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:11,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:11,003 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:11,004 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:54:11,003 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:11,003 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:54:11,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:11,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:11,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:11,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionDeletedRowsDaughter' 2023-04-22 14:54:11,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:54:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:11,003 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:54:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:11,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:11,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:54:11,004 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionDeletedRowsDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:54:11,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:11,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:11,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:11,004 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:54:11,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:54:11,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:11,005 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:11,005 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:54:11,005 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:11,005 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:11,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:54:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:54:11,104 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:54:11,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:54:11,108 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:11,109 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:11,110 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:11,110 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:11,110 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=72, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:11,112 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=001cf566a5015e8ef1438c27e2e9c824, UNASSIGN}] 2023-04-22 14:54:11,112 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=73, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=001cf566a5015e8ef1438c27e2e9c824, UNASSIGN 2023-04-22 14:54:11,113 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=001cf566a5015e8ef1438c27e2e9c824, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:11,113 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1682175251113"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175251113"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175251113"}]},"ts":"1682175251113"} 2023-04-22 14:54:11,114 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=73, state=RUNNABLE; CloseRegionProcedure 001cf566a5015e8ef1438c27e2e9c824, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:54:11,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 001cf566a5015e8ef1438c27e2e9c824, disabling compactions & flushes 2023-04-22 14:54:11,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:11,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:11,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. after waiting 0 ms 2023-04-22 14:54:11,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:11,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 001cf566a5015e8ef1438c27e2e9c824 2/2 column families, dataSize=50 B heapSize=720 B 2023-04-22 14:54:11,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:11,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:11,288 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=25 B at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:11,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:11,293 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf1/55357e0c72ac4abb88c4a27636ec7c3e as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:11,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:11,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/55357e0c72ac4abb88c4a27636ec7c3e, entries=1, sequenceid=103, filesize=4.9 K 2023-04-22 14:54:11,297 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.tmp/cf2/814cbc7d4e2e4fe3bd7d7dca927e9c1a as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:11,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:11,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/814cbc7d4e2e4fe3bd7d7dca927e9c1a, entries=1, sequenceid=103, filesize=4.9 K 2023-04-22 14:54:11,303 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~50 B/50, heapSize ~688 B/688, currentSize=0 B/0 for 001cf566a5015e8ef1438c27e2e9c824 in 37ms, sequenceid=103, compaction requested=true 2023-04-22 14:54:11,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=1 2023-04-22 14:54:11,309 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:54:11,310 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824. 2023-04-22 14:54:11,310 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 001cf566a5015e8ef1438c27e2e9c824: 2023-04-22 14:54:11,312 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,313 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=73 updating hbase:meta row=001cf566a5015e8ef1438c27e2e9c824, regionState=CLOSED 2023-04-22 14:54:11,313 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1682175251313"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175251313"}]},"ts":"1682175251313"} 2023-04-22 14:54:11,316 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=73 2023-04-22 14:54:11,316 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=73, state=SUCCESS; CloseRegionProcedure 001cf566a5015e8ef1438c27e2e9c824, server=jenkins-hbase4.apache.org,33597,1682175218631 in 200 msec 2023-04-22 14:54:11,318 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=72 2023-04-22 14:54:11,318 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=001cf566a5015e8ef1438c27e2e9c824, UNASSIGN in 204 msec 2023-04-22 14:54:11,331 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=72 splitting 20 storefiles, region=001cf566a5015e8ef1438c27e2e9c824, threads=16 2023-04-22 14:54:11,331 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/06f106ab43e844379d17015d273cf8e9 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,331 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,331 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/255122a7d4ac44e191a32667e4e2c68b for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,331 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,332 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,332 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,332 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/814cbc7d4e2e4fe3bd7d7dca927e9c1a for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,332 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e6d6d24cf4e343e5b9e33a8b6f8e3b62 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,339 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/800f49944858458aba0cbdf9ea0ab3cb for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,339 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/ff0bfaf3fe824d1ca8a37799277b9aba for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,339 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/368ef33579cc4830990add57e6c07fba for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,339 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/55357e0c72ac4abb88c4a27636ec7c3e for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,340 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5b4a88346b974583baa74fea26cee27d for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,340 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5d0e1730080c447fa7aad847a9b73b10 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,342 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/89df11190dda482aaeff39606d5c1c39 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,342 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,346 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5d0e1730080c447fa7aad847a9b73b10, top=false 2023-04-22 14:54:11,348 INFO [StoreFileSplitter-pool-6] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:11,348 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/ff0bfaf3fe824d1ca8a37799277b9aba, top=false 2023-04-22 14:54:11,351 INFO [StoreFileSplitter-pool-12] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:11,351 INFO [StoreFileSplitter-pool-7] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e29f83f11c3248a28793be2b209de5ef 2023-04-22 14:54:11,352 INFO [StoreFileSplitter-pool-10] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:11,352 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/255122a7d4ac44e191a32667e4e2c68b, top=false 2023-04-22 14:54:11,353 INFO [StoreFileSplitter-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:11,355 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:11,357 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:11,357 DEBUG [StoreFileSplitter-pool-14] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/89df11190dda482aaeff39606d5c1c39, top=false 2023-04-22 14:54:11,360 INFO [StoreFileSplitter-pool-12] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:11,360 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5b4a88346b974583baa74fea26cee27d, top=true 2023-04-22 14:54:11,361 INFO [StoreFileSplitter-pool-6] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:11,361 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/814cbc7d4e2e4fe3bd7d7dca927e9c1a, top=true 2023-04-22 14:54:11,366 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e6d6d24cf4e343e5b9e33a8b6f8e3b62, top=false 2023-04-22 14:54:11,374 INFO [StoreFileSplitter-pool-10] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:11,374 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/368ef33579cc4830990add57e6c07fba, top=true 2023-04-22 14:54:11,379 INFO [StoreFileSplitter-pool-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:11,379 INFO [StoreFileSplitter-pool-11] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:11,379 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/06f106ab43e844379d17015d273cf8e9, top=true 2023-04-22 14:54:11,379 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/55357e0c72ac4abb88c4a27636ec7c3e, top=true 2023-04-22 14:54:11,379 INFO [StoreFileSplitter-pool-5] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:11,379 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(700): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/800f49944858458aba0cbdf9ea0ab3cb, top=true 2023-04-22 14:54:11,391 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-ff0bfaf3fe824d1ca8a37799277b9aba for child: d6c36ebab54e40f448712ec2278adb5d, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,394 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/ff0bfaf3fe824d1ca8a37799277b9aba for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,394 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,398 INFO [StoreFileSplitter-pool-9] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b8c54a96354d4677a10eece13e6be12c 2023-04-22 14:54:11,407 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-255122a7d4ac44e191a32667e4e2c68b for child: d6c36ebab54e40f448712ec2278adb5d, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,410 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/255122a7d4ac44e191a32667e4e2c68b for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,410 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c1ed0410d24a40f4ae9c7db7c0fe1893 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,411 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5d0e1730080c447fa7aad847a9b73b10 for child: d6c36ebab54e40f448712ec2278adb5d, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,412 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-814cbc7d4e2e4fe3bd7d7dca927e9c1a for child: 302b059e322b8518325c4f182c8edf68, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,412 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/814cbc7d4e2e4fe3bd7d7dca927e9c1a for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,412 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-55357e0c72ac4abb88c4a27636ec7c3e for child: 302b059e322b8518325c4f182c8edf68, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,412 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,412 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5b4a88346b974583baa74fea26cee27d for child: 302b059e322b8518325c4f182c8edf68, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,412 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/55357e0c72ac4abb88c4a27636ec7c3e for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,412 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5b4a88346b974583baa74fea26cee27d for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,412 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=72 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,414 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c1ed0410d24a40f4ae9c7db7c0fe1893, top=false 2023-04-22 14:54:11,416 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5d0e1730080c447fa7aad847a9b73b10 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,432 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-06f106ab43e844379d17015d273cf8e9 for child: 302b059e322b8518325c4f182c8edf68, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,432 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/06f106ab43e844379d17015d273cf8e9 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,433 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-368ef33579cc4830990add57e6c07fba for child: 302b059e322b8518325c4f182c8edf68, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,433 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/368ef33579cc4830990add57e6c07fba for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,433 INFO [StoreFileSplitter-pool-14] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-89df11190dda482aaeff39606d5c1c39 for child: d6c36ebab54e40f448712ec2278adb5d, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,434 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-800f49944858458aba0cbdf9ea0ab3cb for child: 302b059e322b8518325c4f182c8edf68, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,434 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/800f49944858458aba0cbdf9ea0ab3cb for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,438 DEBUG [StoreFileSplitter-pool-14] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/89df11190dda482aaeff39606d5c1c39 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,440 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-e6d6d24cf4e343e5b9e33a8b6f8e3b62 for child: d6c36ebab54e40f448712ec2278adb5d, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,446 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e6d6d24cf4e343e5b9e33a8b6f8e3b62 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,455 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/.splits/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-c1ed0410d24a40f4ae9c7db7c0fe1893 for child: d6c36ebab54e40f448712ec2278adb5d, parent: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,459 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c1ed0410d24a40f4ae9c7db7c0fe1893 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,470 INFO [StoreFileSplitter-pool-7] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e29f83f11c3248a28793be2b209de5ef 2023-04-22 14:54:11,492 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,510 INFO [StoreFileSplitter-pool-9] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b8c54a96354d4677a10eece13e6be12c 2023-04-22 14:54:11,519 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,522 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,525 DEBUG [StoreFileSplitter-pool-15] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,528 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,533 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,533 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,538 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=72 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312 for region: 001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,538 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=72 split storefiles for region 001cf566a5015e8ef1438c27e2e9c824 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-22 14:54:11,586 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-04-22 14:54:11,589 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/recovered.edits/106.seqid, newMaxSeqId=106, maxSeqId=-1 2023-04-22 14:54:11,590 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.","families":{"info":[{"qualifier":"regioninfo","vlen":73,"tag":[],"timestamp":"1682175251590"},{"qualifier":"splitA","vlen":75,"tag":[],"timestamp":"1682175251590"},{"qualifier":"splitB","vlen":75,"tag":[],"timestamp":"1682175251590"}]},"ts":"1682175251590"} 2023-04-22 14:54:11,591 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175251590"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175251590"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175251590"}]},"ts":"1682175251590"} 2023-04-22 14:54:11,591 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175251590"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175251590"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175251590"}]},"ts":"1682175251590"} 2023-04-22 14:54:11,594 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=d6c36ebab54e40f448712ec2278adb5d, ASSIGN}, {pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=302b059e322b8518325c4f182c8edf68, ASSIGN}] 2023-04-22 14:54:11,595 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=302b059e322b8518325c4f182c8edf68, ASSIGN 2023-04-22 14:54:11,595 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=76, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=302b059e322b8518325c4f182c8edf68, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,33597,1682175218631; forceNewPlan=false, retain=false 2023-04-22 14:54:11,596 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=d6c36ebab54e40f448712ec2278adb5d, ASSIGN 2023-04-22 14:54:11,597 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=75, ppid=72, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=d6c36ebab54e40f448712ec2278adb5d, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,33597,1682175218631; forceNewPlan=false, retain=false 2023-04-22 14:54:11,746 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-22 14:54:11,748 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=d6c36ebab54e40f448712ec2278adb5d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:11,748 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175251748"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175251748"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175251748"}]},"ts":"1682175251748"} 2023-04-22 14:54:11,749 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=75, state=RUNNABLE; OpenRegionProcedure d6c36ebab54e40f448712ec2278adb5d, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:54:11,752 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=302b059e322b8518325c4f182c8edf68, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:11,752 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175251751"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175251751"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175251751"}]},"ts":"1682175251751"} 2023-04-22 14:54:11,753 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=78, ppid=76, state=RUNNABLE; OpenRegionProcedure 302b059e322b8518325c4f182c8edf68, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:54:11,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:11,904 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d6c36ebab54e40f448712ec2278adb5d, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.', STARTKEY => '', ENDKEY => '60'} 2023-04-22 14:54:11,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:11,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:11,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:11,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:11,905 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:11,906 INFO [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:11,907 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1 2023-04-22 14:54:11,907 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1 2023-04-22 14:54:11,907 INFO [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6c36ebab54e40f448712ec2278adb5d columnFamilyName cf1 2023-04-22 14:54:11,918 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553-bottom 2023-04-22 14:54:11,922 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,923 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c-bottom 2023-04-22 14:54:11,928 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312-bottom 2023-04-22 14:54:11,932 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e-bottom 2023-04-22 14:54:11,936 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5d0e1730080c447fa7aad847a9b73b10 2023-04-22 14:54:11,939 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-89df11190dda482aaeff39606d5c1c39 2023-04-22 14:54:11,942 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-c1ed0410d24a40f4ae9c7db7c0fe1893 2023-04-22 14:54:11,942 INFO [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(310): Store=d6c36ebab54e40f448712ec2278adb5d/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:11,943 INFO [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:11,944 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2 2023-04-22 14:54:11,944 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2 2023-04-22 14:54:11,944 INFO [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d6c36ebab54e40f448712ec2278adb5d columnFamilyName cf2 2023-04-22 14:54:11,956 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de-bottom 2023-04-22 14:54:11,961 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3-bottom 2023-04-22 14:54:11,966 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7-bottom 2023-04-22 14:54:11,969 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:11,971 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef-bottom 2023-04-22 14:54:11,975 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-255122a7d4ac44e191a32667e4e2c68b 2023-04-22 14:54:11,978 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-e6d6d24cf4e343e5b9e33a8b6f8e3b62 2023-04-22 14:54:11,982 DEBUG [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-ff0bfaf3fe824d1ca8a37799277b9aba 2023-04-22 14:54:11,982 INFO [StoreOpener-d6c36ebab54e40f448712ec2278adb5d-1] regionserver.HStore(310): Store=d6c36ebab54e40f448712ec2278adb5d/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:11,983 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:11,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:11,986 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:54:11,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:11,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d6c36ebab54e40f448712ec2278adb5d; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62724874, jitterRate=-0.06532654166221619}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:54:11,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d6c36ebab54e40f448712ec2278adb5d: 2023-04-22 14:54:11,989 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d., pid=77, masterSystemTime=1682175251901 2023-04-22 14:54:11,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:11,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:11,991 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:54:11,991 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:54:11,991 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:11,991 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1912): d6c36ebab54e40f448712ec2278adb5d/cf1 is initiating minor compaction (all files) 2023-04-22 14:54:11,991 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:11,991 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1912): d6c36ebab54e40f448712ec2278adb5d/cf2 is initiating minor compaction (all files) 2023-04-22 14:54:11,991 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6c36ebab54e40f448712ec2278adb5d/cf1 in testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:11,991 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6c36ebab54e40f448712ec2278adb5d/cf2 in testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:11,991 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5d0e1730080c447fa7aad847a9b73b10, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-c1ed0410d24a40f4ae9c7db7c0fe1893, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-89df11190dda482aaeff39606d5c1c39, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/.tmp, totalSize=35.0 K 2023-04-22 14:54:11,991 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-ff0bfaf3fe824d1ca8a37799277b9aba, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-e6d6d24cf4e343e5b9e33a8b6f8e3b62, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-255122a7d4ac44e191a32667e4e2c68b, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/.tmp, totalSize=35.0 K 2023-04-22 14:54:11,992 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting 8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175249436 2023-04-22 14:54:11,992 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting 24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175249436 2023-04-22 14:54:11,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:11,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:11,992 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:11,992 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 302b059e322b8518325c4f182c8edf68, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.', STARTKEY => '60', ENDKEY => ''} 2023-04-22 14:54:11,992 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175249631 2023-04-22 14:54:11,992 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting 39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175249631 2023-04-22 14:54:11,993 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=75 updating hbase:meta row=d6c36ebab54e40f448712ec2278adb5d, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:11,993 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175249844 2023-04-22 14:54:11,993 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting 6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175249844 2023-04-22 14:54:11,993 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175251993"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175251993"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175251993"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175251993"}]},"ts":"1682175251993"} 2023-04-22 14:54:11,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:11,993 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionDeletedRowsDaughter 302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:11,993 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-ff0bfaf3fe824d1ca8a37799277b9aba, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175250027 2023-04-22 14:54:11,993 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5d0e1730080c447fa7aad847a9b73b10, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175250027 2023-04-22 14:54:11,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:11,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:11,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:11,994 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-e6d6d24cf4e343e5b9e33a8b6f8e3b62, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175250205 2023-04-22 14:54:11,995 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-c1ed0410d24a40f4ae9c7db7c0fe1893, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175250205 2023-04-22 14:54:11,995 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-255122a7d4ac44e191a32667e4e2c68b, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175250400 2023-04-22 14:54:11,996 INFO [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:11,996 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-89df11190dda482aaeff39606d5c1c39, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175250400 2023-04-22 14:54:11,996 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175250582 2023-04-22 14:54:11,996 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=77, resume processing ppid=75 2023-04-22 14:54:11,996 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175250582 2023-04-22 14:54:11,996 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=75, state=SUCCESS; OpenRegionProcedure d6c36ebab54e40f448712ec2278adb5d, server=jenkins-hbase4.apache.org,33597,1682175218631 in 245 msec 2023-04-22 14:54:11,997 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1 2023-04-22 14:54:11,997 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1 2023-04-22 14:54:11,997 INFO [StoreOpener-302b059e322b8518325c4f182c8edf68-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 302b059e322b8518325c4f182c8edf68 columnFamilyName cf1 2023-04-22 14:54:11,998 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=d6c36ebab54e40f448712ec2278adb5d, ASSIGN in 403 msec 2023-04-22 14:54:12,009 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553-top 2023-04-22 14:54:12,012 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6c36ebab54e40f448712ec2278adb5d#cf1#compaction#72 average throughput is 1.40 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:12,012 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] throttle.PressureAwareThroughputController(145): d6c36ebab54e40f448712ec2278adb5d#cf2#compaction#73 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:12,018 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,018 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c-top 2023-04-22 14:54:12,021 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312-top 2023-04-22 14:54:12,025 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e-top 2023-04-22 14:54:12,029 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:12,029 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:12,029 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/.tmp/cf1/6f2f3d989a904e7f94d56d7e268164dc as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/6f2f3d989a904e7f94d56d7e268164dc 2023-04-22 14:54:12,030 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/.tmp/cf2/b2a136f445b74597a6ac94d798eaeb80 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/b2a136f445b74597a6ac94d798eaeb80 2023-04-22 14:54:12,032 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:12,033 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:12,034 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in d6c36ebab54e40f448712ec2278adb5d/cf2 of d6c36ebab54e40f448712ec2278adb5d into b2a136f445b74597a6ac94d798eaeb80(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:12,034 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6c36ebab54e40f448712ec2278adb5d: 2023-04-22 14:54:12,034 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d., storeName=d6c36ebab54e40f448712ec2278adb5d/cf2, priority=9, startTime=1682175251989; duration=0sec 2023-04-22 14:54:12,034 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in d6c36ebab54e40f448712ec2278adb5d/cf1 of d6c36ebab54e40f448712ec2278adb5d into 6f2f3d989a904e7f94d56d7e268164dc(size=6.8 K), total size for store is 6.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:12,035 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:12,035 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6c36ebab54e40f448712ec2278adb5d: 2023-04-22 14:54:12,035 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d., storeName=d6c36ebab54e40f448712ec2278adb5d/cf1, priority=9, startTime=1682175251989; duration=0sec 2023-04-22 14:54:12,035 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:12,036 INFO [StoreFileOpener-cf1-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:12,036 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:12,036 INFO [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(310): Store=302b059e322b8518325c4f182c8edf68/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:12,036 INFO [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:12,037 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2 2023-04-22 14:54:12,037 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2 2023-04-22 14:54:12,037 INFO [StoreOpener-302b059e322b8518325c4f182c8edf68-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 302b059e322b8518325c4f182c8edf68 columnFamilyName cf2 2023-04-22 14:54:12,045 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de-top 2023-04-22 14:54:12,048 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3-top 2023-04-22 14:54:12,051 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7-top 2023-04-22 14:54:12,055 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,055 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef-top 2023-04-22 14:54:12,059 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:12,059 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:12,063 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:12,063 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:12,068 INFO [StoreFileOpener-cf2-1] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:12,068 DEBUG [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:12,068 INFO [StoreOpener-302b059e322b8518325c4f182c8edf68-1] regionserver.HStore(310): Store=302b059e322b8518325c4f182c8edf68/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:12,069 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:12,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:12,073 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionDeletedRowsDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:54:12,074 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:12,075 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 302b059e322b8518325c4f182c8edf68; next sequenceid=107; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59851495, jitterRate=-0.10814322531223297}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:54:12,075 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 302b059e322b8518325c4f182c8edf68: 2023-04-22 14:54:12,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68., pid=78, masterSystemTime=1682175251901 2023-04-22 14:54:12,076 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:12,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:12,077 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:54:12,077 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:54:12,077 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,077 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1912): 302b059e322b8518325c4f182c8edf68/cf1 is initiating minor compaction (all files) 2023-04-22 14:54:12,077 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,077 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1912): 302b059e322b8518325c4f182c8edf68/cf2 is initiating minor compaction (all files) 2023-04-22 14:54:12,077 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 302b059e322b8518325c4f182c8edf68/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,077 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 302b059e322b8518325c4f182c8edf68/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,078 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-06f106ab43e844379d17015d273cf8e9, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-800f49944858458aba0cbdf9ea0ab3cb, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-814cbc7d4e2e4fe3bd7d7dca927e9c1a] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/.tmp, totalSize=34.8 K 2023-04-22 14:54:12,078 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-368ef33579cc4830990add57e6c07fba, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5b4a88346b974583baa74fea26cee27d, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-55357e0c72ac4abb88c4a27636ec7c3e] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/.tmp, totalSize=34.8 K 2023-04-22 14:54:12,078 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting 24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175249436 2023-04-22 14:54:12,078 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting 8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175249436 2023-04-22 14:54:12,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,079 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting 39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175249631 2023-04-22 14:54:12,079 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175249631 2023-04-22 14:54:12,079 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting 6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175249844 2023-04-22 14:54:12,079 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=76 updating hbase:meta row=302b059e322b8518325c4f182c8edf68, regionState=OPEN, openSeqNum=107, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:12,080 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175249844 2023-04-22 14:54:12,080 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682175250582 2023-04-22 14:54:12,080 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175252079"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175252079"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175252079"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175252079"}]},"ts":"1682175252079"} 2023-04-22 14:54:12,080 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824, keycount=4, bloomtype=ROW, size=5.1 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1682175250582 2023-04-22 14:54:12,080 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-06f106ab43e844379d17015d273cf8e9, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-04-22 14:54:12,080 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-368ef33579cc4830990add57e6c07fba, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=9223372036854775807 2023-04-22 14:54:12,080 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-800f49944858458aba0cbdf9ea0ab3cb, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-04-22 14:54:12,081 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-814cbc7d4e2e4fe3bd7d7dca927e9c1a, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-04-22 14:54:12,081 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5b4a88346b974583baa74fea26cee27d, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=99, earliestPutTs=9223372036854775807 2023-04-22 14:54:12,081 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-55357e0c72ac4abb88c4a27636ec7c3e, keycount=1, bloomtype=ROW, size=4.9 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-04-22 14:54:12,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=78, resume processing ppid=76 2023-04-22 14:54:12,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; OpenRegionProcedure 302b059e322b8518325c4f182c8edf68, server=jenkins-hbase4.apache.org,33597,1682175218631 in 328 msec 2023-04-22 14:54:12,085 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=76, resume processing ppid=72 2023-04-22 14:54:12,085 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=76, ppid=72, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=302b059e322b8518325c4f182c8edf68, ASSIGN in 489 msec 2023-04-22 14:54:12,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 in 977 msec 2023-04-22 14:54:12,098 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] throttle.PressureAwareThroughputController(145): 302b059e322b8518325c4f182c8edf68#cf2#compaction#74 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:12,099 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] throttle.PressureAwareThroughputController(145): 302b059e322b8518325c4f182c8edf68#cf1#compaction#75 average throughput is 0.00 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:12,115 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:54:12,117 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 7728f7362d134785885fb0946bcca3cc: NONE, but ROW specified in column family configuration 2023-04-22 14:54:12,118 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/.tmp/cf2/7728f7362d134785885fb0946bcca3cc as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/7728f7362d134785885fb0946bcca3cc 2023-04-22 14:54:12,123 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 7728f7362d134785885fb0946bcca3cc: NONE, but ROW specified in column family configuration 2023-04-22 14:54:12,123 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 302b059e322b8518325c4f182c8edf68/cf2 of 302b059e322b8518325c4f182c8edf68 into 7728f7362d134785885fb0946bcca3cc(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:12,124 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 302b059e322b8518325c4f182c8edf68: 2023-04-22 14:54:12,124 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68., storeName=302b059e322b8518325c4f182c8edf68/cf2, priority=9, startTime=1682175252077; duration=0sec 2023-04-22 14:54:12,124 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:12,128 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 87006a1b5b144340b32ceb4be798b588: NONE, but ROW specified in column family configuration 2023-04-22 14:54:12,129 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/.tmp/cf1/87006a1b5b144340b32ceb4be798b588 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/87006a1b5b144340b32ceb4be798b588 2023-04-22 14:54:12,133 DEBUG [Listener at localhost/43403] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-22 14:54:12,134 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 87006a1b5b144340b32ceb4be798b588: NONE, but ROW specified in column family configuration 2023-04-22 14:54:12,138 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 302b059e322b8518325c4f182c8edf68/cf1 of 302b059e322b8518325c4f182c8edf68 into 87006a1b5b144340b32ceb4be798b588(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:12,138 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 302b059e322b8518325c4f182c8edf68: 2023-04-22 14:54:12,138 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68., storeName=302b059e322b8518325c4f182c8edf68/cf1, priority=9, startTime=1682175252076; duration=0sec 2023-04-22 14:54:12,138 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:12,138 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39430, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-22 14:54:12,138 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33597] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:12,139 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33597] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-22 14:54:12,139 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33597] regionserver.HStore(1912): d6c36ebab54e40f448712ec2278adb5d/cf1 is initiating major compaction (all files) 2023-04-22 14:54:12,140 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33597] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@65631895; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:12,140 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d6c36ebab54e40f448712ec2278adb5d/cf1 in testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:12,140 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33597] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-22 14:54:12,140 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33597] regionserver.HStore(1912): d6c36ebab54e40f448712ec2278adb5d/cf2 is initiating major compaction (all files) 2023-04-22 14:54:12,140 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/6f2f3d989a904e7f94d56d7e268164dc] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/.tmp, totalSize=6.8 K 2023-04-22 14:54:12,140 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d6c36ebab54e40f448712ec2278adb5d/cf2 in testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:12,140 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33597] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@69b2bd67; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:12,140 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/b2a136f445b74597a6ac94d798eaeb80] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/.tmp, totalSize=6.8 K 2023-04-22 14:54:12,140 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting 6f2f3d989a904e7f94d56d7e268164dc, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175249436 2023-04-22 14:54:12,140 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting b2a136f445b74597a6ac94d798eaeb80, keycount=49, bloomtype=ROW, size=6.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175249436 2023-04-22 14:54:12,142 INFO [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33597] regionserver.RSRpcServices(1662): Compacting testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,142 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33597] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-22 14:54:12,142 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33597] regionserver.HStore(1912): 302b059e322b8518325c4f182c8edf68/cf1 is initiating major compaction (all files) 2023-04-22 14:54:12,142 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33597] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@43df2e2c; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:12,142 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33597] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-04-22 14:54:12,142 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33597] regionserver.HStore(1912): 302b059e322b8518325c4f182c8edf68/cf2 is initiating major compaction (all files) 2023-04-22 14:54:12,143 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33597] regionserver.CompactSplit(385): Small Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@18cf7fc6; Because: User-triggered major compaction for region testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 14:54:12,144 INFO [Listener at localhost/43403] hbase.Waiter(180): Waiting up to [6,000] milli-secs(wait.for.ratio=[1]) 2023-04-22 14:54:12,157 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] throttle.PressureAwareThroughputController(145): d6c36ebab54e40f448712ec2278adb5d#cf2#compaction#77 average throughput is 1.40 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:12,157 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] throttle.PressureAwareThroughputController(145): d6c36ebab54e40f448712ec2278adb5d#cf1#compaction#76 average throughput is 1.40 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:12,165 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33597] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. is not mergeable because it has references 2023-04-22 14:54:12,166 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33597] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. is not mergeable because it has references 2023-04-22 14:54:12,186 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/.tmp/cf1/e6710a0ce5f14060a061bd7bf9875378 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/e6710a0ce5f14060a061bd7bf9875378 2023-04-22 14:54:12,188 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/.tmp/cf2/9232a31c7011423ba952d4e5e10cb5b9 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/9232a31c7011423ba952d4e5e10cb5b9 2023-04-22 14:54:12,192 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in d6c36ebab54e40f448712ec2278adb5d/cf1 of d6c36ebab54e40f448712ec2278adb5d into e6710a0ce5f14060a061bd7bf9875378(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:12,192 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6c36ebab54e40f448712ec2278adb5d: 2023-04-22 14:54:12,193 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in d6c36ebab54e40f448712ec2278adb5d/cf2 of d6c36ebab54e40f448712ec2278adb5d into 9232a31c7011423ba952d4e5e10cb5b9(size=6.9 K), total size for store is 6.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:12,193 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d6c36ebab54e40f448712ec2278adb5d: 2023-04-22 14:54:12,194 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d., storeName=cf2, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1682175252140; duration=0sec 2023-04-22 14:54:12,194 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d., storeName=cf1, fileCount=1, fileSize=6.8 K (6.8 K), priority=1, time=1682175252139; duration=0sec 2023-04-22 14:54:12,194 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-22 14:54:12,194 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:54:12,194 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 302b059e322b8518325c4f182c8edf68/cf1 in testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,194 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 302b059e322b8518325c4f182c8edf68/cf2 in testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,194 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/87006a1b5b144340b32ceb4be798b588] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/.tmp, totalSize=5.0 K 2023-04-22 14:54:12,194 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/7728f7362d134785885fb0946bcca3cc] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/.tmp, totalSize=5.0 K 2023-04-22 14:54:12,195 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting 87006a1b5b144340b32ceb4be798b588, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-04-22 14:54:12,195 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting 7728f7362d134785885fb0946bcca3cc, keycount=0, bloomtype=NONE, size=5.0 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=9223372036854775807 2023-04-22 14:54:12,198 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] throttle.PressureAwareThroughputController(145): 302b059e322b8518325c4f182c8edf68#cf2#compaction#78 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:12,198 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] throttle.PressureAwareThroughputController(145): 302b059e322b8518325c4f182c8edf68#cf1#compaction#79 average throughput is NaN MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:54:12,226 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 6763d92c147f48b7a78c557610499cca: NONE, but ROW specified in column family configuration 2023-04-22 14:54:12,227 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/.tmp/cf2/6763d92c147f48b7a78c557610499cca as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/6763d92c147f48b7a78c557610499cca 2023-04-22 14:54:12,228 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 5b0a587bb8d942b5ad2bfbbf9062834a: NONE, but ROW specified in column family configuration 2023-04-22 14:54:12,229 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/.tmp/cf1/5b0a587bb8d942b5ad2bfbbf9062834a as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/5b0a587bb8d942b5ad2bfbbf9062834a 2023-04-22 14:54:12,232 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 6763d92c147f48b7a78c557610499cca: NONE, but ROW specified in column family configuration 2023-04-22 14:54:12,233 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in 302b059e322b8518325c4f182c8edf68/cf2 of 302b059e322b8518325c4f182c8edf68 into 6763d92c147f48b7a78c557610499cca(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:12,233 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 302b059e322b8518325c4f182c8edf68: 2023-04-22 14:54:12,233 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68., storeName=cf2, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1682175252142; duration=0sec 2023-04-22 14:54:12,233 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:12,234 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStoreFile(437): HFile Bloom filter type for 5b0a587bb8d942b5ad2bfbbf9062834a: NONE, but ROW specified in column family configuration 2023-04-22 14:54:12,235 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1652): Completed major compaction of 1 (all) file(s) in 302b059e322b8518325c4f182c8edf68/cf1 of 302b059e322b8518325c4f182c8edf68 into 5b0a587bb8d942b5ad2bfbbf9062834a(size=5.0 K), total size for store is 5.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:54:12,235 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 302b059e322b8518325c4f182c8edf68: 2023-04-22 14:54:12,235 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction Request=regionName=testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68., storeName=cf1, fileCount=1, fileSize=5.0 K (5.0 K), priority=1, time=1682175252142; duration=0sec 2023-04-22 14:54:12,235 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:54:12,275 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=33597] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. is not mergeable because it has references 2023-04-22 14:54:12,275 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=33597] regionserver.HRegion(1451): Region testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. is not mergeable because it has references 2023-04-22 14:54:12,281 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '60', inclusive 2023-04-22 14:54:12,283 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:54:12,284 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=79, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-04-22 14:54:12,288 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175252288"}]},"ts":"1682175252288"} 2023-04-22 14:54:12,289 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLING in hbase:meta 2023-04-22 14:54:12,292 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLING 2023-04-22 14:54:12,293 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=d6c36ebab54e40f448712ec2278adb5d, UNASSIGN}, {pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=302b059e322b8518325c4f182c8edf68, UNASSIGN}] 2023-04-22 14:54:12,293 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=302b059e322b8518325c4f182c8edf68, UNASSIGN 2023-04-22 14:54:12,294 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=302b059e322b8518325c4f182c8edf68, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:12,294 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175252294"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175252294"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175252294"}]},"ts":"1682175252294"} 2023-04-22 14:54:12,295 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=82, ppid=81, state=RUNNABLE; CloseRegionProcedure 302b059e322b8518325c4f182c8edf68, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:54:12,297 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=79, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=d6c36ebab54e40f448712ec2278adb5d, UNASSIGN 2023-04-22 14:54:12,298 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=d6c36ebab54e40f448712ec2278adb5d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:12,298 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175252298"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175252298"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175252298"}]},"ts":"1682175252298"} 2023-04-22 14:54:12,299 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=80, state=RUNNABLE; CloseRegionProcedure d6c36ebab54e40f448712ec2278adb5d, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:54:12,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-04-22 14:54:12,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:12,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 302b059e322b8518325c4f182c8edf68, disabling compactions & flushes 2023-04-22 14:54:12,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. after waiting 0 ms 2023-04-22 14:54:12,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,448 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-368ef33579cc4830990add57e6c07fba, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5b4a88346b974583baa74fea26cee27d, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/87006a1b5b144340b32ceb4be798b588, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-55357e0c72ac4abb88c4a27636ec7c3e] to archive 2023-04-22 14:54:12,449 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:12,451 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,452 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,453 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,454 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,455 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-368ef33579cc4830990add57e6c07fba to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:12,456 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5b4a88346b974583baa74fea26cee27d to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:12,457 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/87006a1b5b144340b32ceb4be798b588 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/87006a1b5b144340b32ceb4be798b588 2023-04-22 14:54:12,458 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-55357e0c72ac4abb88c4a27636ec7c3e to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:12,459 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-06f106ab43e844379d17015d273cf8e9, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-800f49944858458aba0cbdf9ea0ab3cb, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/7728f7362d134785885fb0946bcca3cc, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-814cbc7d4e2e4fe3bd7d7dca927e9c1a] to archive 2023-04-22 14:54:12,460 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:12,461 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,462 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,463 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,464 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,465 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-06f106ab43e844379d17015d273cf8e9 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:12,466 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-800f49944858458aba0cbdf9ea0ab3cb to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:12,467 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/7728f7362d134785885fb0946bcca3cc to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/7728f7362d134785885fb0946bcca3cc 2023-04-22 14:54:12,468 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-814cbc7d4e2e4fe3bd7d7dca927e9c1a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:12,471 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-04-22 14:54:12,472 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:54:12,473 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68. 2023-04-22 14:54:12,473 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 302b059e322b8518325c4f182c8edf68: 2023-04-22 14:54:12,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:12,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:12,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d6c36ebab54e40f448712ec2278adb5d, disabling compactions & flushes 2023-04-22 14:54:12,474 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:12,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:12,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. after waiting 0 ms 2023-04-22 14:54:12,474 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:12,475 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=302b059e322b8518325c4f182c8edf68, regionState=CLOSED 2023-04-22 14:54:12,475 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175252475"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175252475"}]},"ts":"1682175252475"} 2023-04-22 14:54:12,475 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5d0e1730080c447fa7aad847a9b73b10, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-c1ed0410d24a40f4ae9c7db7c0fe1893, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-89df11190dda482aaeff39606d5c1c39, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/6f2f3d989a904e7f94d56d7e268164dc, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c-bottom] to archive 2023-04-22 14:54:12,476 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:12,478 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/8b161201a2364c3eba9b9cf63655c553.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,478 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=82, resume processing ppid=81 2023-04-22 14:54:12,478 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=81, state=SUCCESS; CloseRegionProcedure 302b059e322b8518325c4f182c8edf68, server=jenkins-hbase4.apache.org,33597,1682175218631 in 181 msec 2023-04-22 14:54:12,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=302b059e322b8518325c4f182c8edf68, UNASSIGN in 185 msec 2023-04-22 14:54:12,479 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,480 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/c5eb2af66eca4eee9ea2de5e97ad9312.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,481 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5d0e1730080c447fa7aad847a9b73b10 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-5d0e1730080c447fa7aad847a9b73b10 2023-04-22 14:54:12,482 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-c1ed0410d24a40f4ae9c7db7c0fe1893 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-c1ed0410d24a40f4ae9c7db7c0fe1893 2023-04-22 14:54:12,483 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-89df11190dda482aaeff39606d5c1c39 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-89df11190dda482aaeff39606d5c1c39 2023-04-22 14:54:12,484 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/6f2f3d989a904e7f94d56d7e268164dc to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/6f2f3d989a904e7f94d56d7e268164dc 2023-04-22 14:54:12,485 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/b8c54a96354d4677a10eece13e6be12c.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,486 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-ff0bfaf3fe824d1ca8a37799277b9aba, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-e6d6d24cf4e343e5b9e33a8b6f8e3b62, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-255122a7d4ac44e191a32667e4e2c68b, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/b2a136f445b74597a6ac94d798eaeb80, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef-bottom] to archive 2023-04-22 14:54:12,487 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:54:12,488 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/24bf657f679f4277a86b89e4687e82de.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,489 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/39955a28897d41b7af911cc88a9a00f3.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,490 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/6ffc2df596cf4051a3d18ba09abfbdc7.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,491 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-ff0bfaf3fe824d1ca8a37799277b9aba to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-ff0bfaf3fe824d1ca8a37799277b9aba 2023-04-22 14:54:12,492 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-e6d6d24cf4e343e5b9e33a8b6f8e3b62 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-e6d6d24cf4e343e5b9e33a8b6f8e3b62 2023-04-22 14:54:12,493 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-255122a7d4ac44e191a32667e4e2c68b to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/testSplitTableRegionDeletedRowsDaughter=001cf566a5015e8ef1438c27e2e9c824-255122a7d4ac44e191a32667e4e2c68b 2023-04-22 14:54:12,494 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/b2a136f445b74597a6ac94d798eaeb80 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/b2a136f445b74597a6ac94d798eaeb80 2023-04-22 14:54:12,495 DEBUG [StoreCloser-testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/e29f83f11c3248a28793be2b209de5ef.001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/recovered.edits/113.seqid, newMaxSeqId=113, maxSeqId=106 2023-04-22 14:54:12,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:54:12,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d. 2023-04-22 14:54:12,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d6c36ebab54e40f448712ec2278adb5d: 2023-04-22 14:54:12,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:12,501 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=d6c36ebab54e40f448712ec2278adb5d, regionState=CLOSED 2023-04-22 14:54:12,501 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.","families":{"info":[{"qualifier":"regioninfo","vlen":75,"tag":[],"timestamp":"1682175252501"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175252501"}]},"ts":"1682175252501"} 2023-04-22 14:54:12,504 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=80 2023-04-22 14:54:12,504 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=80, state=SUCCESS; CloseRegionProcedure d6c36ebab54e40f448712ec2278adb5d, server=jenkins-hbase4.apache.org,33597,1682175218631 in 203 msec 2023-04-22 14:54:12,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=80, resume processing ppid=79 2023-04-22 14:54:12,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=79, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionDeletedRowsDaughter, region=d6c36ebab54e40f448712ec2278adb5d, UNASSIGN in 211 msec 2023-04-22 14:54:12,506 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175252506"}]},"ts":"1682175252506"} 2023-04-22 14:54:12,507 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionDeletedRowsDaughter, state=DISABLED in hbase:meta 2023-04-22 14:54:12,508 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionDeletedRowsDaughter to state=DISABLED 2023-04-22 14:54:12,509 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 223 msec 2023-04-22 14:54:12,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=79 2023-04-22 14:54:12,590 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 79 completed 2023-04-22 14:54:12,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,596 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=84, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=84 2023-04-22 14:54:12,598 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=84, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,603 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,603 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:12,603 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:12,606 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/recovered.edits] 2023-04-22 14:54:12,606 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/recovered.edits] 2023-04-22 14:54:12,606 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/recovered.edits] 2023-04-22 14:54:12,610 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/5b0a587bb8d942b5ad2bfbbf9062834a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf1/5b0a587bb8d942b5ad2bfbbf9062834a 2023-04-22 14:54:12,610 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/e6710a0ce5f14060a061bd7bf9875378 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf1/e6710a0ce5f14060a061bd7bf9875378 2023-04-22 14:54:12,610 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/368ef33579cc4830990add57e6c07fba to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/368ef33579cc4830990add57e6c07fba 2023-04-22 14:54:12,612 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/55357e0c72ac4abb88c4a27636ec7c3e to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/55357e0c72ac4abb88c4a27636ec7c3e 2023-04-22 14:54:12,613 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/6763d92c147f48b7a78c557610499cca to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/cf2/6763d92c147f48b7a78c557610499cca 2023-04-22 14:54:12,614 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/9232a31c7011423ba952d4e5e10cb5b9 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/cf2/9232a31c7011423ba952d4e5e10cb5b9 2023-04-22 14:54:12,614 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5b4a88346b974583baa74fea26cee27d to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5b4a88346b974583baa74fea26cee27d 2023-04-22 14:54:12,616 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5d0e1730080c447fa7aad847a9b73b10 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/5d0e1730080c447fa7aad847a9b73b10 2023-04-22 14:54:12,618 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/recovered.edits/113.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68/recovered.edits/113.seqid 2023-04-22 14:54:12,618 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/recovered.edits/113.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d/recovered.edits/113.seqid 2023-04-22 14:54:12,618 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/89df11190dda482aaeff39606d5c1c39 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/89df11190dda482aaeff39606d5c1c39 2023-04-22 14:54:12,618 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/302b059e322b8518325c4f182c8edf68 2023-04-22 14:54:12,620 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/d6c36ebab54e40f448712ec2278adb5d 2023-04-22 14:54:12,622 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/8b161201a2364c3eba9b9cf63655c553 2023-04-22 14:54:12,624 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/b8c54a96354d4677a10eece13e6be12c 2023-04-22 14:54:12,625 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c1ed0410d24a40f4ae9c7db7c0fe1893 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c1ed0410d24a40f4ae9c7db7c0fe1893 2023-04-22 14:54:12,629 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/c5eb2af66eca4eee9ea2de5e97ad9312 2023-04-22 14:54:12,630 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf1/d67d47c8dfb84d7cbb2bf4272b8fb59e 2023-04-22 14:54:12,632 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/06f106ab43e844379d17015d273cf8e9 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/06f106ab43e844379d17015d273cf8e9 2023-04-22 14:54:12,634 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/24bf657f679f4277a86b89e4687e82de 2023-04-22 14:54:12,635 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/255122a7d4ac44e191a32667e4e2c68b to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/255122a7d4ac44e191a32667e4e2c68b 2023-04-22 14:54:12,636 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/39955a28897d41b7af911cc88a9a00f3 2023-04-22 14:54:12,638 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/6ffc2df596cf4051a3d18ba09abfbdc7 2023-04-22 14:54:12,639 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/800f49944858458aba0cbdf9ea0ab3cb to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/800f49944858458aba0cbdf9ea0ab3cb 2023-04-22 14:54:12,641 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/814cbc7d4e2e4fe3bd7d7dca927e9c1a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/814cbc7d4e2e4fe3bd7d7dca927e9c1a 2023-04-22 14:54:12,645 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e29f83f11c3248a28793be2b209de5ef 2023-04-22 14:54:12,646 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e6d6d24cf4e343e5b9e33a8b6f8e3b62 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/e6d6d24cf4e343e5b9e33a8b6f8e3b62 2023-04-22 14:54:12,647 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/ff0bfaf3fe824d1ca8a37799277b9aba to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/cf2/ff0bfaf3fe824d1ca8a37799277b9aba 2023-04-22 14:54:12,649 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/recovered.edits/106.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824/recovered.edits/106.seqid 2023-04-22 14:54:12,650 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionDeletedRowsDaughter/001cf566a5015e8ef1438c27e2e9c824 2023-04-22 14:54:12,650 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionDeletedRowsDaughter regions 2023-04-22 14:54:12,653 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=84, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,655 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionDeletedRowsDaughter from hbase:meta 2023-04-22 14:54:12,657 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionDeletedRowsDaughter' descriptor. 2023-04-22 14:54:12,658 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=84, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,658 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionDeletedRowsDaughter' from region states. 2023-04-22 14:54:12,658 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175252658"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:12,658 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175252658"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:12,658 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175252658"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:12,660 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-22 14:54:12,660 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 001cf566a5015e8ef1438c27e2e9c824, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1682175248926.001cf566a5015e8ef1438c27e2e9c824.', STARTKEY => '', ENDKEY => ''}, {ENCODED => d6c36ebab54e40f448712ec2278adb5d, NAME => 'testSplitTableRegionDeletedRowsDaughter,,1682175251108.d6c36ebab54e40f448712ec2278adb5d.', STARTKEY => '', ENDKEY => '60'}, {ENCODED => 302b059e322b8518325c4f182c8edf68, NAME => 'testSplitTableRegionDeletedRowsDaughter,60,1682175251108.302b059e322b8518325c4f182c8edf68.', STARTKEY => '60', ENDKEY => ''}] 2023-04-22 14:54:12,660 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionDeletedRowsDaughter' as deleted. 2023-04-22 14:54:12,661 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionDeletedRowsDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175252660"}]},"ts":"9223372036854775807"} 2023-04-22 14:54:12,662 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionDeletedRowsDaughter state from META 2023-04-22 14:54:12,664 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=84, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:54:12,665 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter in 73 msec 2023-04-22 14:54:12,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=84 2023-04-22 14:54:12,699 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionDeletedRowsDaughter, procId: 84 completed 2023-04-22 14:54:12,717 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionDeletedRowsDaughter Thread=411 (was 402) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:58720 [Waiting for operation #14] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:48074 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:47604 [Waiting for operation #26] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:50110 [Waiting for operation #56] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:56502 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:48114 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:56798 [Waiting for operation #12] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:48088 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:57052 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:56744 [Waiting for operation #33] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:2;jenkins-hbase4:33597-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:56838 [Waiting for operation #9] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:48102 [Waiting for operation #4] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:48082 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:59304 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:40048 [Waiting for operation #55] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6454e058-metaLookup-shared--pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_422801511_15 at /127.0.0.1:58912 [Waiting for operation #10] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=868 (was 860) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=422 (was 450), ProcessCount=172 (was 172), AvailableMemoryMB=12543 (was 12595) 2023-04-22 14:54:12,734 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=411, OpenFileDescriptor=868, MaxFileDescriptor=60000, SystemLoadAverage=422, ProcessCount=172, AvailableMemoryMB=12542 2023-04-22 14:54:12,734 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:54:12,737 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:54:12,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:54:12,739 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:54:12,741 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:54:12,742 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:54:12,743 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:12,744 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36 empty. 2023-04-22 14:54:12,744 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:12,744 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-04-22 14:54:12,755 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-04-22 14:54:12,756 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(7675): creating {ENCODED => be1c77199f735729a41e26e3a9a83c36, NAME => 'testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:54:12,763 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:12,763 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1604): Closing be1c77199f735729a41e26e3a9a83c36, disabling compactions & flushes 2023-04-22 14:54:12,764 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:12,764 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:12,764 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. after waiting 0 ms 2023-04-22 14:54:12,764 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:12,764 INFO [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:12,764 DEBUG [RegionOpenAndInit-testRollbackForSplitTableRegionWithReplica-pool-0] regionserver.HRegion(1558): Region close journal for be1c77199f735729a41e26e3a9a83c36: 2023-04-22 14:54:12,766 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:54:12,766 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175252766"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175252766"}]},"ts":"1682175252766"} 2023-04-22 14:54:12,768 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:54:12,768 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:54:12,768 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175252768"}]},"ts":"1682175252768"} 2023-04-22 14:54:12,769 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLING in hbase:meta 2023-04-22 14:54:12,772 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:54:12,773 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:54:12,773 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:54:12,773 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:54:12,773 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:54:12,773 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, ASSIGN}] 2023-04-22 14:54:12,774 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, ASSIGN 2023-04-22 14:54:12,775 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=86, ppid=85, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35417,1682175218529; forceNewPlan=false, retain=false 2023-04-22 14:54:12,925 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:54:12,928 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=be1c77199f735729a41e26e3a9a83c36, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:12,928 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175252928"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175252928"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175252928"}]},"ts":"1682175252928"} 2023-04-22 14:54:12,929 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=86, state=RUNNABLE; OpenRegionProcedure be1c77199f735729a41e26e3a9a83c36, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:54:13,083 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be1c77199f735729a41e26e3a9a83c36, NAME => 'testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:54:13,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:13,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:13,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,085 INFO [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,087 DEBUG [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/cf1 2023-04-22 14:54:13,087 DEBUG [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/cf1 2023-04-22 14:54:13,087 INFO [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be1c77199f735729a41e26e3a9a83c36 columnFamilyName cf1 2023-04-22 14:54:13,088 INFO [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] regionserver.HStore(310): Store=be1c77199f735729a41e26e3a9a83c36/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:13,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,091 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:54:13,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened be1c77199f735729a41e26e3a9a83c36; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62696002, jitterRate=-0.06575676798820496}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-22 14:54:13,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for be1c77199f735729a41e26e3a9a83c36: 2023-04-22 14:54:13,094 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36., pid=87, masterSystemTime=1682175253081 2023-04-22 14:54:13,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,096 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,096 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=86 updating hbase:meta row=be1c77199f735729a41e26e3a9a83c36, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:13,097 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175253096"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175253096"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175253096"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175253096"}]},"ts":"1682175253096"} 2023-04-22 14:54:13,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=86 2023-04-22 14:54:13,100 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=86, state=SUCCESS; OpenRegionProcedure be1c77199f735729a41e26e3a9a83c36, server=jenkins-hbase4.apache.org,35417,1682175218529 in 169 msec 2023-04-22 14:54:13,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=85 2023-04-22 14:54:13,101 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=85, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, ASSIGN in 327 msec 2023-04-22 14:54:13,102 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:54:13,102 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175253102"}]},"ts":"1682175253102"} 2023-04-22 14:54:13,103 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=ENABLED in hbase:meta 2023-04-22 14:54:13,110 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=85, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:54:13,111 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica in 372 msec 2023-04-22 14:54:13,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table testRollbackForSplitTableRegionWithReplica from 'testRollbackForSplitTableRegionWithReplica', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'testRollbackForSplitTableRegionWithReplica', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '2'}}, {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-22 14:54:13,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=88, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:54:13,298 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-04-22 14:54:13,299 DEBUG [PEWorker-1] util.FSTableDescriptors(519): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000001 2023-04-22 14:54:13,299 INFO [PEWorker-1] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/.tabledesc/.tableinfo.0000000002 2023-04-22 14:54:13,310 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=89, ppid=88, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica}] 2023-04-22 14:54:13,337 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, REOPEN/MOVE}] 2023-04-22 14:54:13,338 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, REOPEN/MOVE 2023-04-22 14:54:13,339 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=be1c77199f735729a41e26e3a9a83c36, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:13,339 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175253339"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175253339"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175253339"}]},"ts":"1682175253339"} 2023-04-22 14:54:13,341 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=91, ppid=90, state=RUNNABLE; CloseRegionProcedure be1c77199f735729a41e26e3a9a83c36, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:54:13,497 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing be1c77199f735729a41e26e3a9a83c36, disabling compactions & flushes 2023-04-22 14:54:13,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. after waiting 0 ms 2023-04-22 14:54:13,499 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-22 14:54:13,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:54:13,509 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for be1c77199f735729a41e26e3a9a83c36: 2023-04-22 14:54:13,509 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: be1c77199f735729a41e26e3a9a83c36 to self. 2023-04-22 14:54:13,517 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,517 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=be1c77199f735729a41e26e3a9a83c36, regionState=CLOSED 2023-04-22 14:54:13,518 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175253517"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175253517"}]},"ts":"1682175253517"} 2023-04-22 14:54:13,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=91, resume processing ppid=90 2023-04-22 14:54:13,525 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=91, ppid=90, state=SUCCESS; CloseRegionProcedure be1c77199f735729a41e26e3a9a83c36, server=jenkins-hbase4.apache.org,35417,1682175218529 in 179 msec 2023-04-22 14:54:13,527 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=90, ppid=89, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35417,1682175218529; forceNewPlan=false, retain=true 2023-04-22 14:54:13,677 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:54:13,677 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=be1c77199f735729a41e26e3a9a83c36, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:13,678 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175253677"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175253677"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175253677"}]},"ts":"1682175253677"} 2023-04-22 14:54:13,682 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=92, ppid=90, state=RUNNABLE; OpenRegionProcedure be1c77199f735729a41e26e3a9a83c36, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:54:13,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => be1c77199f735729a41e26e3a9a83c36, NAME => 'testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:54:13,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:13,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:13,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,839 INFO [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,840 DEBUG [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/cf1 2023-04-22 14:54:13,840 DEBUG [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/cf1 2023-04-22 14:54:13,841 INFO [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region be1c77199f735729a41e26e3a9a83c36 columnFamilyName cf1 2023-04-22 14:54:13,842 INFO [StoreOpener-be1c77199f735729a41e26e3a9a83c36-1] regionserver.HStore(310): Store=be1c77199f735729a41e26e3a9a83c36/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:54:13,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:54:13,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened be1c77199f735729a41e26e3a9a83c36; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64469982, jitterRate=-0.03932240605354309}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-22 14:54:13,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for be1c77199f735729a41e26e3a9a83c36: 2023-04-22 14:54:13,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36., pid=92, masterSystemTime=1682175253833 2023-04-22 14:54:13,850 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,850 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:54:13,850 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=90 updating hbase:meta row=be1c77199f735729a41e26e3a9a83c36, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:54:13,850 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175253850"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175253850"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175253850"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175253850"}]},"ts":"1682175253850"} 2023-04-22 14:54:13,855 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=92, resume processing ppid=90 2023-04-22 14:54:13,855 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=92, ppid=90, state=SUCCESS; OpenRegionProcedure be1c77199f735729a41e26e3a9a83c36, server=jenkins-hbase4.apache.org,35417,1682175218529 in 170 msec 2023-04-22 14:54:13,857 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=90, resume processing ppid=89 2023-04-22 14:54:13,858 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=90, ppid=89, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, REOPEN/MOVE in 518 msec 2023-04-22 14:54:13,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=89, resume processing ppid=88 2023-04-22 14:54:13,861 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=89, ppid=88, state=SUCCESS; ReopenTableRegionsProcedure table=testRollbackForSplitTableRegionWithReplica in 550 msec 2023-04-22 14:54:13,869 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b, ASSIGN}] 2023-04-22 14:54:13,871 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b, ASSIGN 2023-04-22 14:54:13,871 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=88, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-22 14:54:14,022 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:54:14,023 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:54:14,023 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:54:14,023 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:54:14,023 DEBUG [jenkins-hbase4:43497] balancer.BaseLoadBalancer$Cluster(378): Number of tables=3, number of hosts=1, number of racks=1 2023-04-22 14:54:14,026 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=436aea79dba9499e96e168de6ff0b85b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:54:14,026 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175254026"},{"qualifier":"sn_0001","vlen":45,"tag":[],"timestamp":"1682175254026"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1682175254026"}]},"ts":"1682175254026"} 2023-04-22 14:54:14,031 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=94, ppid=93, state=RUNNABLE; OpenRegionProcedure 436aea79dba9499e96e168de6ff0b85b, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:54:14,187 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b. 2023-04-22 14:54:14,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 436aea79dba9499e96e168de6ff0b85b, NAME => 'testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1} 2023-04-22 14:54:14,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:54:14,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRollbackForSplitTableRegionWithReplica 436aea79dba9499e96e168de6ff0b85b 2023-04-22 14:54:14,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:54:14,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 436aea79dba9499e96e168de6ff0b85b 2023-04-22 14:54:14,188 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 436aea79dba9499e96e168de6ff0b85b 2023-04-22 14:54:14,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:14,289 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:14,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:14,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:14,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:14,690 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:14,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:14,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:14,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:15,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,095 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,291 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRollbackForSplitTableRegionWithReplica' 2023-04-22 14:54:16,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,496 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,697 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:16,998 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:17,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:17,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:17,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:17,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:17,407 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:54:17,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:17,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:17,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:17,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:17,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:18,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,203 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,604 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:19,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,005 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,105 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,506 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,606 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,706 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:20,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,007 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,107 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,207 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,308 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,508 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,608 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,809 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:21,909 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,009 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,109 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,210 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,310 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,510 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,610 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,711 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:22,911 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,011 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,112 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,212 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,513 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,613 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,713 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,813 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:23,913 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,214 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,615 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,715 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,815 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:24,916 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,116 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,216 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,517 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,617 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,717 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:25,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,018 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,319 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,419 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,619 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,720 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:26,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,622 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,822 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:27,923 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,223 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,424 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:28,925 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,226 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,326 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,626 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:29,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,228 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,629 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:30,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,531 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:31,932 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,132 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,733 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:32,934 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,034 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,134 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,234 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,335 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,435 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,535 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,836 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:33,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,036 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,136 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,237 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,437 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,537 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,738 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,838 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:34,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,038 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,138 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,239 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,339 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,539 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,639 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,740 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,840 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:35,940 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,040 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,141 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,241 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,541 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,642 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:36,942 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,042 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,143 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,243 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,343 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,443 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,544 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,644 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,744 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:37,944 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,145 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,245 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,546 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,646 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,746 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,846 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:38,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,047 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,147 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,347 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,448 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,548 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,648 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,748 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,848 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:39,949 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,049 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,094 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-22 14:54:40,129 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-22 14:54:40,149 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,153 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-22 14:54:40,249 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,350 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,450 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,650 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,751 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,851 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:40,951 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,051 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,151 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,252 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,352 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,452 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,552 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,653 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,753 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,853 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:41,953 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,053 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,154 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,454 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,555 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,655 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,755 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,855 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:42,956 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,056 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,156 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,357 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,457 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,557 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,757 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,858 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:43,958 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,058 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,158 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,259 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,359 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,459 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,559 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,760 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,860 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:44,960 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,161 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,461 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,562 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,662 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,762 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,862 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:45,962 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,063 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,163 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,263 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,463 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,564 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,664 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,764 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,864 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:46,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,065 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,165 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,265 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,466 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,666 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,766 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,867 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:47,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,067 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,167 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,268 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,368 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,568 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,668 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,769 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,869 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:48,969 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,069 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,170 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,370 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,571 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,671 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:49,971 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,072 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,172 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,372 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,473 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,573 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,673 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:50,974 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,074 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,174 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,375 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,676 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,876 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:51,976 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,076 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,377 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,578 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,678 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,878 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:52,979 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,079 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,379 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,480 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,580 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,680 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,780 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,881 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:53,981 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,081 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,382 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,482 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,582 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,682 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,783 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,883 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:54,983 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,689 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,790 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,890 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:55,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,291 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,391 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,491 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,592 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,692 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,892 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:56,993 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,193 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,494 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,594 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,694 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,895 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:57,995 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,095 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,196 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,496 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,597 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,697 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:58,998 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:59,098 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:59,198 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:59,298 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:59,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:59,499 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:59,599 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:59,699 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:59,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:54:59,900 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,000 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,100 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,200 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,301 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,501 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:00,902 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,002 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,202 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,603 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:01,904 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,204 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,305 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,405 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,505 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,605 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,705 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,806 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:02,906 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,307 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,507 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,607 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:03,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,008 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,108 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,209 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,309 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,409 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,609 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,710 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:04,910 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,110 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,211 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,311 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,511 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,611 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,712 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:05,912 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,012 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,213 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,513 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,714 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,814 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:06,914 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,014 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,215 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,515 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,616 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,716 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,816 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:07,916 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,016 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,217 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,317 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,417 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,518 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:08,918 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,019 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,219 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,319 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,419 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,519 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,620 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,720 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:09,920 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,020 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,221 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,321 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,421 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,522 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,622 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,722 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,822 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:10,922 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,123 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,223 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,524 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,624 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,724 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,824 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:11,925 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,025 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,125 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,225 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,325 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,426 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,526 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,626 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,726 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:12,927 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,027 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,127 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,227 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,328 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-22 14:55:13,428 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-22 14:55:13,528 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,628 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-22 14:55:13,729 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,929 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:13,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-22 14:55:14,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:14,130 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:14,230 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:14,330 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:14,430 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:14,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=88 2023-04-22 14:55:14,530 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:14,631 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:14,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:14,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:14,931 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:15,032 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:15,132 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:15,232 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:15,332 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:15,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:15,472 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:15,473 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:15,474 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:15,474 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:15,474 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:15,476 ERROR [PEWorker-1] assignment.SplitTableRegionProcedure(347): Splitting be1c77199f735729a41e26e3a9a83c36, pid=95, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up at org.apache.hadoop.hbase.master.assignment.AssignmentManagerUtil.createUnassignProceduresForSplitOrMerge(AssignmentManagerUtil.java:120) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.createUnassignProcedures(SplitTableRegionProcedure.java:881) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:298) at org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:94) at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188) at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1646) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:15,480 INFO [PEWorker-1] procedure2.ProcedureExecutor(1528): Rolled back pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 exec-time=7 msec 2023-04-22 14:55:15,533 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:15,633 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro(132): Slow down replica region open a bit 2023-04-22 14:55:15,734 INFO [StoreOpener-436aea79dba9499e96e168de6ff0b85b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 436aea79dba9499e96e168de6ff0b85b 2023-04-22 14:55:15,738 DEBUG [StoreOpener-436aea79dba9499e96e168de6ff0b85b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/cf1 2023-04-22 14:55:15,738 DEBUG [StoreOpener-436aea79dba9499e96e168de6ff0b85b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/cf1 2023-04-22 14:55:15,739 INFO [StoreOpener-436aea79dba9499e96e168de6ff0b85b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 436aea79dba9499e96e168de6ff0b85b columnFamilyName cf1 2023-04-22 14:55:15,740 INFO [StoreOpener-436aea79dba9499e96e168de6ff0b85b-1] regionserver.HStore(310): Store=436aea79dba9499e96e168de6ff0b85b/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:15,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 436aea79dba9499e96e168de6ff0b85b; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72905075, jitterRate=0.08637027442455292}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-22 14:55:15,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 436aea79dba9499e96e168de6ff0b85b: 2023-04-22 14:55:15,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b., pid=94, masterSystemTime=1682175254184 2023-04-22 14:55:15,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b. 2023-04-22 14:55:15,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b. 2023-04-22 14:55:15,744 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=436aea79dba9499e96e168de6ff0b85b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:15,744 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175315744"},{"qualifier":"server_0001","vlen":31,"tag":[],"timestamp":"1682175315744"},{"qualifier":"serverstartcode_0001","vlen":8,"tag":[],"timestamp":"1682175315744"},{"qualifier":"seqnumDuringOpen_0001","vlen":8,"tag":[],"timestamp":"1682175315744"}]},"ts":"1682175315744"} 2023-04-22 14:55:15,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=94, resume processing ppid=93 2023-04-22 14:55:15,748 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=93, state=SUCCESS; OpenRegionProcedure 436aea79dba9499e96e168de6ff0b85b, server=jenkins-hbase4.apache.org,33597,1682175218631 in 1 mins, 1.718 sec 2023-04-22 14:55:15,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=93, resume processing ppid=88 2023-04-22 14:55:15,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=88, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b, ASSIGN in 1 mins, 1.879 sec 2023-04-22 14:55:15,754 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica in 1 mins, 2.499 sec 2023-04-22 14:55:15,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] ipc.RpcServer(528): (responseTooSlow): {"call":"ModifyTable(org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest)","starttimems":"1682175253247","responsesize":"2","method":"ModifyTable","param":"TODO: class org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$ModifyTableRequest","processingtimems":62508,"client":"172.31.14.131:51926","queuetimems":0,"class":"HMaster"} 2023-04-22 14:55:15,756 DEBUG [RPCClient-NioEventLoopGroup-6-2] ipc.NettyRpcDuplexHandler(156): Unknown callId: 605, skipping over this response of 3 bytes 2023-04-22 14:55:17,724 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(335): procId=95 exception: org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up 2023-04-22 14:55:17,726 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:55:17,728 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:17,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:17,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=96, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:17,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-04-22 14:55:17,732 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175317732"}]},"ts":"1682175317732"} 2023-04-22 14:55:17,733 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLING in hbase:meta 2023-04-22 14:55:17,735 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRollbackForSplitTableRegionWithReplica to state=DISABLING 2023-04-22 14:55:17,735 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, UNASSIGN}, {pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b, UNASSIGN}] 2023-04-22 14:55:17,736 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b, UNASSIGN 2023-04-22 14:55:17,737 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=436aea79dba9499e96e168de6ff0b85b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:17,737 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175317737"},{"qualifier":"sn_0001","vlen":45,"tag":[],"timestamp":"1682175317737"},{"qualifier":"state_0001","vlen":7,"tag":[],"timestamp":"1682175317737"}]},"ts":"1682175317737"} 2023-04-22 14:55:17,738 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=98, state=RUNNABLE; CloseRegionProcedure 436aea79dba9499e96e168de6ff0b85b, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:55:17,741 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=96, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, UNASSIGN 2023-04-22 14:55:17,741 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=be1c77199f735729a41e26e3a9a83c36, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:17,742 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175317741"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175317741"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175317741"}]},"ts":"1682175317741"} 2023-04-22 14:55:17,744 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=97, state=RUNNABLE; CloseRegionProcedure be1c77199f735729a41e26e3a9a83c36, server=jenkins-hbase4.apache.org,35417,1682175218529}] 2023-04-22 14:55:17,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-04-22 14:55:17,890 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 436aea79dba9499e96e168de6ff0b85b 2023-04-22 14:55:17,891 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 436aea79dba9499e96e168de6ff0b85b, disabling compactions & flushes 2023-04-22 14:55:17,892 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b. 2023-04-22 14:55:17,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b. 2023-04-22 14:55:17,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b. after waiting 0 ms 2023-04-22 14:55:17,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b. 2023-04-22 14:55:17,892 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:17,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b. 2023-04-22 14:55:17,895 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 436aea79dba9499e96e168de6ff0b85b: 2023-04-22 14:55:17,895 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:55:17,896 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing be1c77199f735729a41e26e3a9a83c36, disabling compactions & flushes 2023-04-22 14:55:17,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:55:17,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:55:17,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. after waiting 0 ms 2023-04-22 14:55:17,897 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:55:17,898 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 436aea79dba9499e96e168de6ff0b85b 2023-04-22 14:55:17,902 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=436aea79dba9499e96e168de6ff0b85b, regionState=CLOSED 2023-04-22 14:55:17,903 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175317902"},{"qualifier":"state_0001","vlen":6,"tag":[],"timestamp":"1682175317902"}]},"ts":"1682175317902"} 2023-04-22 14:55:17,906 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=98 2023-04-22 14:55:17,906 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=98, state=SUCCESS; CloseRegionProcedure 436aea79dba9499e96e168de6ff0b85b, server=jenkins-hbase4.apache.org,33597,1682175218631 in 166 msec 2023-04-22 14:55:17,907 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b, UNASSIGN in 171 msec 2023-04-22 14:55:17,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/recovered.edits/7.seqid, newMaxSeqId=7, maxSeqId=4 2023-04-22 14:55:17,909 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:17,910 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36. 2023-04-22 14:55:17,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for be1c77199f735729a41e26e3a9a83c36: 2023-04-22 14:55:17,911 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:55:17,912 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=be1c77199f735729a41e26e3a9a83c36, regionState=CLOSED 2023-04-22 14:55:17,912 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"regioninfo","vlen":76,"tag":[],"timestamp":"1682175317912"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175317912"}]},"ts":"1682175317912"} 2023-04-22 14:55:17,916 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=97 2023-04-22 14:55:17,916 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=97, state=SUCCESS; CloseRegionProcedure be1c77199f735729a41e26e3a9a83c36, server=jenkins-hbase4.apache.org,35417,1682175218529 in 171 msec 2023-04-22 14:55:17,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=97, resume processing ppid=96 2023-04-22 14:55:17,919 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=96, state=SUCCESS; TransitRegionStateProcedure table=testRollbackForSplitTableRegionWithReplica, region=be1c77199f735729a41e26e3a9a83c36, UNASSIGN in 181 msec 2023-04-22 14:55:17,922 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175317922"}]},"ts":"1682175317922"} 2023-04-22 14:55:17,924 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRollbackForSplitTableRegionWithReplica, state=DISABLED in hbase:meta 2023-04-22 14:55:17,926 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRollbackForSplitTableRegionWithReplica to state=DISABLED 2023-04-22 14:55:17,928 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica in 198 msec 2023-04-22 14:55:18,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=96 2023-04-22 14:55:18,035 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 96 completed 2023-04-22 14:55:18,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:18,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:18,038 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=101, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:18,039 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=101, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:18,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=101 2023-04-22 14:55:18,046 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:55:18,048 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/recovered.edits] 2023-04-22 14:55:18,054 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/recovered.edits/7.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36/recovered.edits/7.seqid 2023-04-22 14:55:18,054 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRollbackForSplitTableRegionWithReplica/be1c77199f735729a41e26e3a9a83c36 2023-04-22 14:55:18,054 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRollbackForSplitTableRegionWithReplica regions 2023-04-22 14:55:18,056 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=101, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:18,060 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testRollbackForSplitTableRegionWithReplica from hbase:meta 2023-04-22 14:55:18,062 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRollbackForSplitTableRegionWithReplica' descriptor. 2023-04-22 14:55:18,063 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=101, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:18,063 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRollbackForSplitTableRegionWithReplica' from region states. 2023-04-22 14:55:18,064 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175318063"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:18,064 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175318063"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:18,066 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 2 regions from META 2023-04-22 14:55:18,066 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => be1c77199f735729a41e26e3a9a83c36, NAME => 'testRollbackForSplitTableRegionWithReplica,,1682175252738.be1c77199f735729a41e26e3a9a83c36.', STARTKEY => '', ENDKEY => ''}, {ENCODED => 436aea79dba9499e96e168de6ff0b85b, NAME => 'testRollbackForSplitTableRegionWithReplica,,1682175252738_0001.436aea79dba9499e96e168de6ff0b85b.', STARTKEY => '', ENDKEY => '', REPLICA_ID => 1}] 2023-04-22 14:55:18,066 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRollbackForSplitTableRegionWithReplica' as deleted. 2023-04-22 14:55:18,067 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRollbackForSplitTableRegionWithReplica","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175318066"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:18,070 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRollbackForSplitTableRegionWithReplica state from META 2023-04-22 14:55:18,072 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=101, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:18,073 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica in 37 msec 2023-04-22 14:55:18,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=101 2023-04-22 14:55:18,140 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRollbackForSplitTableRegionWithReplica, procId: 101 completed 2023-04-22 14:55:18,154 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRollbackForSplitTableRegionWithReplica Thread=345 (was 411), OpenFileDescriptor=655 (was 868), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=343 (was 422), ProcessCount=172 (was 172), AvailableMemoryMB=12428 (was 12542) 2023-04-22 14:55:18,168 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=346, OpenFileDescriptor=655, MaxFileDescriptor=60000, SystemLoadAverage=343, ProcessCount=172, AvailableMemoryMB=12428 2023-04-22 14:55:18,169 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:55:18,172 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:55:18,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:55:18,174 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:18,175 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:55:18,176 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:55:18,178 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,178 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164 empty. 2023-04-22 14:55:18,179 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,179 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-04-22 14:55:18,198 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/.tabledesc/.tableinfo.0000000001 2023-04-22 14:55:18,199 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(7675): creating {ENCODED => 99790a94004fb6fc0bfe1f00f245d164, NAME => 'testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testInvalidSplitKey', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:55:18,234 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(866): Instantiated testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:18,235 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1604): Closing 99790a94004fb6fc0bfe1f00f245d164, disabling compactions & flushes 2023-04-22 14:55:18,235 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1626): Closing region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:18,235 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:18,235 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1714): Acquired close lock on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. after waiting 0 ms 2023-04-22 14:55:18,235 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1724): Updates disabled for region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:18,235 INFO [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1838): Closed testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:18,235 DEBUG [RegionOpenAndInit-testInvalidSplitKey-pool-0] regionserver.HRegion(1558): Region close journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:18,237 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:55:18,238 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1682175318238"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175318238"}]},"ts":"1682175318238"} 2023-04-22 14:55:18,239 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:55:18,240 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:55:18,240 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175318240"}]},"ts":"1682175318240"} 2023-04-22 14:55:18,241 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLING in hbase:meta 2023-04-22 14:55:18,245 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:55:18,246 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:55:18,246 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:55:18,246 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:55:18,247 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:55:18,247 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=99790a94004fb6fc0bfe1f00f245d164, ASSIGN}] 2023-04-22 14:55:18,249 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testInvalidSplitKey, region=99790a94004fb6fc0bfe1f00f245d164, ASSIGN 2023-04-22 14:55:18,250 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=103, ppid=102, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testInvalidSplitKey, region=99790a94004fb6fc0bfe1f00f245d164, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:55:18,400 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:55:18,401 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=99790a94004fb6fc0bfe1f00f245d164, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,401 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1682175318401"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175318401"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175318401"}]},"ts":"1682175318401"} 2023-04-22 14:55:18,403 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=103, state=RUNNABLE; OpenRegionProcedure 99790a94004fb6fc0bfe1f00f245d164, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:18,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:18,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 99790a94004fb6fc0bfe1f00f245d164, NAME => 'testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:55:18,561 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:18,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testInvalidSplitKey 99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:18,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,572 INFO [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,574 DEBUG [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1 2023-04-22 14:55:18,574 DEBUG [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1 2023-04-22 14:55:18,574 INFO [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 99790a94004fb6fc0bfe1f00f245d164 columnFamilyName cf1 2023-04-22 14:55:18,575 INFO [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] regionserver.HStore(310): Store=99790a94004fb6fc0bfe1f00f245d164/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:18,575 INFO [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,576 DEBUG [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2 2023-04-22 14:55:18,576 DEBUG [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2 2023-04-22 14:55:18,576 INFO [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 99790a94004fb6fc0bfe1f00f245d164 columnFamilyName cf2 2023-04-22 14:55:18,577 INFO [StoreOpener-99790a94004fb6fc0bfe1f00f245d164-1] regionserver.HStore(310): Store=99790a94004fb6fc0bfe1f00f245d164/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:18,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testInvalidSplitKey descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:18,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:18,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:55:18,586 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 99790a94004fb6fc0bfe1f00f245d164; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60537600, jitterRate=-0.09791946411132812}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:18,586 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:18,587 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164., pid=104, masterSystemTime=1682175318555 2023-04-22 14:55:18,589 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:18,589 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:18,589 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=103 updating hbase:meta row=99790a94004fb6fc0bfe1f00f245d164, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,589 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1682175318589"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175318589"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175318589"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175318589"}]},"ts":"1682175318589"} 2023-04-22 14:55:18,593 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=103 2023-04-22 14:55:18,593 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=103, state=SUCCESS; OpenRegionProcedure 99790a94004fb6fc0bfe1f00f245d164, server=jenkins-hbase4.apache.org,36267,1682175218583 in 188 msec 2023-04-22 14:55:18,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=102 2023-04-22 14:55:18,596 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=102, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=99790a94004fb6fc0bfe1f00f245d164, ASSIGN in 346 msec 2023-04-22 14:55:18,597 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:55:18,597 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175318597"}]},"ts":"1682175318597"} 2023-04-22 14:55:18,598 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=ENABLED in hbase:meta 2023-04-22 14:55:18,601 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=102, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testInvalidSplitKey execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:55:18,603 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey in 428 msec 2023-04-22 14:55:18,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:18,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-22 14:55:18,692 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-22 14:55:18,692 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,692 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-22 14:55:18,692 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-22 14:55:18,693 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,693 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:18,696 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,696 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,697 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:18,697 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,697 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:18,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:18,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:18,696 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:18,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:18,698 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:18,698 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,699 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:18,699 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:18,699 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:18,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:18,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:18,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:18,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:18,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:18,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,701 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,701 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:18,702 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,702 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,702 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:18,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:18,705 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:18,705 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:18,705 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:18,705 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. started... 2023-04-22 14:55:18,708 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 99790a94004fb6fc0bfe1f00f245d164 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-22 14:55:18,769 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/fce7ad84a178409790dfb08e9105e277 2023-04-22 14:55:18,835 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/bb1f0d3f411b4de8839f377bb9e7ac1b 2023-04-22 14:55:18,841 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/fce7ad84a178409790dfb08e9105e277 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/fce7ad84a178409790dfb08e9105e277 2023-04-22 14:55:18,845 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/fce7ad84a178409790dfb08e9105e277, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:55:18,846 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/bb1f0d3f411b4de8839f377bb9e7ac1b as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/bb1f0d3f411b4de8839f377bb9e7ac1b 2023-04-22 14:55:18,850 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/bb1f0d3f411b4de8839f377bb9e7ac1b, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:55:18,852 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 99790a94004fb6fc0bfe1f00f245d164 in 144ms, sequenceid=6, compaction requested=false 2023-04-22 14:55:18,852 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testInvalidSplitKey' 2023-04-22 14:55:18,852 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:18,853 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:18,855 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,855 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,856 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,856 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:18,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-22 14:55:18,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16185e2d[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:18,858 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-22 14:55:18,858 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:18,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:18,860 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,860 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:18,862 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,863 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:18,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-22 14:55:18,866 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-22 14:55:18,867 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:18,867 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:18,867 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-22 14:55:18,867 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:18,869 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:18,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,869 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,869 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,869 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:18,870 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:18,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,871 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,872 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:18,873 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,874 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:18,874 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:18,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:18,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,878 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:18,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-22 14:55:18,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:18,879 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:18,879 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:18,879 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:18,879 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:18,879 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:18,878 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:18,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:18,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:18,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:18,979 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:18,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:18,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:19,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:19,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-22 14:55:19,011 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-22 14:55:19,011 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,011 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-22 14:55:19,011 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-22 14:55:19,012 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,012 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,014 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,014 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,014 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,014 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,015 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,015 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,017 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,018 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,018 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,018 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:19,018 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:19,018 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:19,018 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. started... 2023-04-22 14:55:19,018 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 99790a94004fb6fc0bfe1f00f245d164 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:19,019 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,023 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,036 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/e59d6f4dbd484aeb8af5f2bb230b76df 2023-04-22 14:55:19,123 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/8a858a53393a45059de5bb28f6e60605 2023-04-22 14:55:19,129 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/e59d6f4dbd484aeb8af5f2bb230b76df as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/e59d6f4dbd484aeb8af5f2bb230b76df 2023-04-22 14:55:19,134 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/e59d6f4dbd484aeb8af5f2bb230b76df, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:55:19,135 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/8a858a53393a45059de5bb28f6e60605 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8a858a53393a45059de5bb28f6e60605 2023-04-22 14:55:19,141 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8a858a53393a45059de5bb28f6e60605, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:55:19,142 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 99790a94004fb6fc0bfe1f00f245d164 in 124ms, sequenceid=19, compaction requested=false 2023-04-22 14:55:19,142 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:19,142 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:19,142 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:19,142 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:19,142 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,142 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,142 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,145 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,145 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,145 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,145 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-22 14:55:19,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2056b297[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:19,148 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-22 14:55:19,148 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,149 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,149 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,149 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,149 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,149 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,150 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,150 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,150 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:19,150 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,150 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,150 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:19,150 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,150 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,150 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:19,151 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,151 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,151 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,151 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,151 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,151 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,152 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,152 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:19,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-22 14:55:19,157 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-22 14:55:19,157 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:19,157 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:19,157 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-22 14:55:19,157 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:19,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,165 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,166 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,167 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,167 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,170 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,170 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,171 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,176 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,176 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,176 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,176 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,176 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,176 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-22 14:55:19,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:19,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,176 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,176 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,177 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,176 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,177 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:19,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,177 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,177 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:19,177 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,178 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,178 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,178 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,178 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,178 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,178 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,278 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:19,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:19,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:19,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:19,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-22 14:55:19,297 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-22 14:55:19,297 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,297 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-22 14:55:19,297 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-22 14:55:19,298 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,298 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,300 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,300 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,300 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,300 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,301 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,301 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,304 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,305 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,307 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:19,312 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:19,313 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:19,313 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. started... 2023-04-22 14:55:19,313 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 99790a94004fb6fc0bfe1f00f245d164 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:19,360 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/c08d7bad86594fbd8766323d9cad9068 2023-04-22 14:55:19,380 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/6c22ef2ae5d94307b5fa1806f2b8f524 2023-04-22 14:55:19,388 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/c08d7bad86594fbd8766323d9cad9068 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/c08d7bad86594fbd8766323d9cad9068 2023-04-22 14:55:19,407 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/c08d7bad86594fbd8766323d9cad9068, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:55:19,408 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/6c22ef2ae5d94307b5fa1806f2b8f524 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/6c22ef2ae5d94307b5fa1806f2b8f524 2023-04-22 14:55:19,418 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/6c22ef2ae5d94307b5fa1806f2b8f524, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:55:19,419 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 99790a94004fb6fc0bfe1f00f245d164 in 106ms, sequenceid=32, compaction requested=true 2023-04-22 14:55:19,419 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:19,419 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:19,419 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:19,420 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:19,420 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,420 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,420 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,422 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,422 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,423 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,423 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-22 14:55:19,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@101d2d47[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:19,425 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-22 14:55:19,425 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,426 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,426 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,426 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,426 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,427 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:19,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:19,427 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,428 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,429 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,429 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,429 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:19,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-22 14:55:19,434 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-22 14:55:19,434 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:19,434 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:19,434 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-22 14:55:19,434 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:19,436 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,436 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,436 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,436 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,436 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,436 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,436 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,437 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,438 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,438 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,441 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,441 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,442 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,454 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-22 14:55:19,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:19,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,455 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:19,461 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:19,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testInvalidSplitKey because node does not exist (not an error) 2023-04-22 14:55:19,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,461 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,462 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,462 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,462 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,462 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,462 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,462 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,561 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:19,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:19,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:19,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:19,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-22 14:55:19,584 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-22 14:55:19,584 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,584 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-22 14:55:19,584 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-22 14:55:19,585 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,585 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,587 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,587 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,587 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,587 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,588 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,588 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,604 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,604 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,604 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,604 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,604 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,606 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,606 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,606 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,612 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,612 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,612 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,616 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,616 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,616 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,616 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,616 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,619 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,619 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,620 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,628 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:19,628 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:19,628 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:19,628 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. started... 2023-04-22 14:55:19,628 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 99790a94004fb6fc0bfe1f00f245d164 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:19,669 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/244cf0f831624599b9b9fb7816be9beb 2023-04-22 14:55:19,707 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/2545531ea90443649dcde47cb1b95193 2023-04-22 14:55:19,713 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/244cf0f831624599b9b9fb7816be9beb as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/244cf0f831624599b9b9fb7816be9beb 2023-04-22 14:55:19,717 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/244cf0f831624599b9b9fb7816be9beb, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:55:19,718 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/2545531ea90443649dcde47cb1b95193 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/2545531ea90443649dcde47cb1b95193 2023-04-22 14:55:19,723 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/2545531ea90443649dcde47cb1b95193, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:55:19,724 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 99790a94004fb6fc0bfe1f00f245d164 in 96ms, sequenceid=45, compaction requested=true 2023-04-22 14:55:19,724 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:19,724 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:19,724 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:19,724 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:19,724 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,724 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,724 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,725 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,725 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,726 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,726 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-22 14:55:19,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23f0a06e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:19,728 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-22 14:55:19,728 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,730 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,730 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,730 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,730 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:19,730 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,730 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:19,730 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,730 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,730 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,731 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,733 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,733 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,733 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,733 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,733 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,733 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,734 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:19,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-22 14:55:19,737 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-22 14:55:19,737 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:19,737 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:19,737 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-22 14:55:19,737 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:19,739 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,739 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,739 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,739 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,740 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,741 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,741 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,742 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,743 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,743 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,744 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,746 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-22 14:55:19,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:19,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,747 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:19,747 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,748 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:19,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testInvalidSplitKey because node does not exist (not an error) 2023-04-22 14:55:19,751 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,751 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,751 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/testInvalidSplitKey because node does not exist (not an error) 2023-04-22 14:55:19,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,848 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:19,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:19,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:19,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:19,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-22 14:55:19,867 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-22 14:55:19,867 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,867 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-22 14:55:19,867 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-22 14:55:19,867 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,867 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,868 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,868 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,868 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,869 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,869 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,869 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:19,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:19,871 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,871 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:19,871 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:19,872 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,872 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. started... 2023-04-22 14:55:19,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:19,872 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 99790a94004fb6fc0bfe1f00f245d164 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,872 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,875 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,875 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,875 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,875 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,875 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,875 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,907 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/d32d9ff37e124c8fa9929bc9304c86b9 2023-04-22 14:55:19,946 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/f153499e57544214aff18322c789e2d8 2023-04-22 14:55:19,952 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/d32d9ff37e124c8fa9929bc9304c86b9 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/d32d9ff37e124c8fa9929bc9304c86b9 2023-04-22 14:55:19,958 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/d32d9ff37e124c8fa9929bc9304c86b9, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:55:19,959 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/f153499e57544214aff18322c789e2d8 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/f153499e57544214aff18322c789e2d8 2023-04-22 14:55:19,965 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/f153499e57544214aff18322c789e2d8, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:55:19,966 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 99790a94004fb6fc0bfe1f00f245d164 in 94ms, sequenceid=58, compaction requested=true 2023-04-22 14:55:19,966 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:19,966 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:19,967 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:19,968 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,968 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,969 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-22 14:55:19,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@774d6b0b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:19,972 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-22 14:55:19,972 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,973 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,974 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,973 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:19,974 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,974 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:19,979 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,979 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,979 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,979 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,983 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,983 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,983 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,984 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:19,984 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,984 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:19,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-22 14:55:19,988 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-22 14:55:19,988 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:19,988 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:19,988 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-22 14:55:19,988 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:19,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:19,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:19,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,993 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:19,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:19,993 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:19,994 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,995 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:19,995 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:19,996 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:19,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:19,999 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,000 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,000 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,000 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,000 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,000 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,000 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,000 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-22 14:55:20,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:20,000 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,001 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:20,001 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:20,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,000 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,001 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,002 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,002 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,002 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,002 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,002 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,002 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,002 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,101 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:20,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:20,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:20,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testInvalidSplitKey was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:20,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testInvalidSplitKey 2023-04-22 14:55:20,120 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testInvalidSplitKey' 2023-04-22 14:55:20,120 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,120 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testInvalidSplitKey' starting 'acquire' 2023-04-22 14:55:20,120 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testInvalidSplitKey', kicking off acquire phase on members. 2023-04-22 14:55:20,121 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,121 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,122 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,122 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,123 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,123 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,123 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,123 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:20,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:20,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:20,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:20,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testInvalidSplitKey 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:20,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testInvalidSplitKey is still present, removing 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:20,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:20,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testInvalidSplitKey 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testInvalidSplitKey' with timeout 60000ms 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:20,125 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:20,126 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:20,126 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testInvalidSplitKey' starting 'acquire' stage 2023-04-22 14:55:20,126 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:20,126 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:20,126 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:20,126 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. started... 2023-04-22 14:55:20,126 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 99790a94004fb6fc0bfe1f00f245d164 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:20,128 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,128 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,128 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,128 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:20,129 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,129 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:20,145 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/6f29960c94d7405095701f87e00bfb51 2023-04-22 14:55:20,163 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/722dd7f9abe94ee28f5eca217882c3cc 2023-04-22 14:55:20,167 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/6f29960c94d7405095701f87e00bfb51 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/6f29960c94d7405095701f87e00bfb51 2023-04-22 14:55:20,175 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/6f29960c94d7405095701f87e00bfb51, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:55:20,176 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/722dd7f9abe94ee28f5eca217882c3cc as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/722dd7f9abe94ee28f5eca217882c3cc 2023-04-22 14:55:20,180 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/722dd7f9abe94ee28f5eca217882c3cc, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:55:20,181 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 99790a94004fb6fc0bfe1f00f245d164 in 55ms, sequenceid=71, compaction requested=true 2023-04-22 14:55:20,181 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:20,181 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:20,181 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:20,181 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:20,181 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testInvalidSplitKey' locally acquired 2023-04-22 14:55:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testInvalidSplitKey) in zk 2023-04-22 14:55:20,183 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,183 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,184 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testInvalidSplitKey' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testInvalidSplitKey' on coordinator 2023-04-22 14:55:20,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@629abd7f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:20,186 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testInvalidSplitKey' starting 'in-barrier' execution. 2023-04-22 14:55:20,186 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,187 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,187 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,187 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,187 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:20,187 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,188 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testInvalidSplitKey' received 'reached' from coordinator. 2023-04-22 14:55:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testInvalidSplitKey' locally completed 2023-04-22 14:55:20,188 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testInvalidSplitKey' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:20,189 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:20,189 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,189 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testInvalidSplitKey' has notified controller of completion 2023-04-22 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,191 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:20,191 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testInvalidSplitKey' completed. 2023-04-22 14:55:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testInvalidSplitKey' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:20,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testInvalidSplitKey', counting down latch. Waiting for 0 more 2023-04-22 14:55:20,194 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testInvalidSplitKey' execution completed 2023-04-22 14:55:20,194 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:20,194 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:20,194 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testInvalidSplitKey 2023-04-22 14:55:20,194 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testInvalidSplitKeyincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:20,196 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,196 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,196 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,196 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,196 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,196 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:20,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,196 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,197 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,197 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:20,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,197 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:20,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,199 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:20,199 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testInvalidSplitKey 2023-04-22 14:55:20,200 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,202 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:20,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testInvalidSplitKey' 2023-04-22 14:55:20,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:20,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,203 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:20,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:20,203 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testInvalidSplitKey'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:20,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:20,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,203 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:20,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,203 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,204 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testInvalidSplitKey 2023-04-22 14:55:20,204 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:20,204 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,204 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:20,204 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,204 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testInvalidSplitKey 2023-04-22 14:55:20,204 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testInvalidSplitKey 2023-04-22 14:55:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:20,304 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:20,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:20,312 INFO [Listener at localhost/43403] assignment.SplitTableRegionProcedure(206): testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. splitKey isn't explicitly specified, will try to find a best split key from RS jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,314 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.StoreUtils(129): cannot split hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/e59d6f4dbd484aeb8af5f2bb230b76df because midkey is the same as first or last row 2023-04-22 14:55:20,314 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.StoreUtils(129): cannot split hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8a858a53393a45059de5bb28f6e60605 because midkey is the same as first or last row 2023-04-22 14:55:20,314 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.HRegion(2745): Flushing 99790a94004fb6fc0bfe1f00f245d164 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-22 14:55:20,329 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/d81bb39725e547c5928e08322cd3d0fc 2023-04-22 14:55:20,342 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/8084ad5236144001a89606a4d13f3987 2023-04-22 14:55:20,347 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf1/d81bb39725e547c5928e08322cd3d0fc as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/d81bb39725e547c5928e08322cd3d0fc 2023-04-22 14:55:20,352 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/d81bb39725e547c5928e08322cd3d0fc, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:55:20,353 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/.tmp/cf2/8084ad5236144001a89606a4d13f3987 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8084ad5236144001a89606a4d13f3987 2023-04-22 14:55:20,363 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8084ad5236144001a89606a4d13f3987, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:55:20,364 INFO [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 99790a94004fb6fc0bfe1f00f245d164 in 50ms, sequenceid=82, compaction requested=true 2023-04-22 14:55:20,365 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.HRegion(2446): Flush status journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:20,365 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.StoreUtils(129): cannot split hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/e59d6f4dbd484aeb8af5f2bb230b76df because midkey is the same as first or last row 2023-04-22 14:55:20,366 DEBUG [RpcServer.priority.RWQ.Fifo.read.handler=1,queue=1,port=36267] regionserver.StoreUtils(129): cannot split hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8a858a53393a45059de5bb28f6e60605 because midkey is the same as first or last row 2023-04-22 14:55:20,366 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,366 DEBUG [Listener at localhost/43403] assignment.TestSplitTableRegionProcedure(438): Expected Split procedure construction failure: Region not splittable because bestSplitPoint = null, maybe table is too small for auto split. For force split, try specifying split row 2023-04-22 14:55:20,366 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:55:20,368 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testInvalidSplitKey 2023-04-22 14:55:20,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testInvalidSplitKey 2023-04-22 14:55:20,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:20,375 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175320374"}]},"ts":"1682175320374"} 2023-04-22 14:55:20,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-04-22 14:55:20,375 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLING in hbase:meta 2023-04-22 14:55:20,377 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testInvalidSplitKey to state=DISABLING 2023-04-22 14:55:20,378 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=99790a94004fb6fc0bfe1f00f245d164, UNASSIGN}] 2023-04-22 14:55:20,379 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testInvalidSplitKey, region=99790a94004fb6fc0bfe1f00f245d164, UNASSIGN 2023-04-22 14:55:20,379 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=99790a94004fb6fc0bfe1f00f245d164, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:20,380 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1682175320379"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175320379"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175320379"}]},"ts":"1682175320379"} 2023-04-22 14:55:20,381 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=107, ppid=106, state=RUNNABLE; CloseRegionProcedure 99790a94004fb6fc0bfe1f00f245d164, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:20,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-04-22 14:55:20,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:20,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 99790a94004fb6fc0bfe1f00f245d164, disabling compactions & flushes 2023-04-22 14:55:20,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:20,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:20,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. after waiting 0 ms 2023-04-22 14:55:20,534 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:20,540 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-22 14:55:20,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:20,541 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164. 2023-04-22 14:55:20,541 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 99790a94004fb6fc0bfe1f00f245d164: 2023-04-22 14:55:20,544 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:20,544 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=99790a94004fb6fc0bfe1f00f245d164, regionState=CLOSED 2023-04-22 14:55:20,545 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.","families":{"info":[{"qualifier":"regioninfo","vlen":53,"tag":[],"timestamp":"1682175320544"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175320544"}]},"ts":"1682175320544"} 2023-04-22 14:55:20,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=107, resume processing ppid=106 2023-04-22 14:55:20,547 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=106, state=SUCCESS; CloseRegionProcedure 99790a94004fb6fc0bfe1f00f245d164, server=jenkins-hbase4.apache.org,36267,1682175218583 in 165 msec 2023-04-22 14:55:20,549 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=106, resume processing ppid=105 2023-04-22 14:55:20,549 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testInvalidSplitKey, region=99790a94004fb6fc0bfe1f00f245d164, UNASSIGN in 169 msec 2023-04-22 14:55:20,549 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175320549"}]},"ts":"1682175320549"} 2023-04-22 14:55:20,551 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testInvalidSplitKey, state=DISABLED in hbase:meta 2023-04-22 14:55:20,552 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testInvalidSplitKey to state=DISABLED 2023-04-22 14:55:20,554 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey in 185 msec 2023-04-22 14:55:20,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-04-22 14:55:20,677 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testInvalidSplitKey, procId: 105 completed 2023-04-22 14:55:20,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testInvalidSplitKey 2023-04-22 14:55:20,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:20,681 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=108, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:20,682 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=108, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:20,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=108 2023-04-22 14:55:20,689 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:20,691 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/recovered.edits] 2023-04-22 14:55:20,697 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/244cf0f831624599b9b9fb7816be9beb to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/244cf0f831624599b9b9fb7816be9beb 2023-04-22 14:55:20,699 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/6f29960c94d7405095701f87e00bfb51 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/6f29960c94d7405095701f87e00bfb51 2023-04-22 14:55:20,700 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/c08d7bad86594fbd8766323d9cad9068 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/c08d7bad86594fbd8766323d9cad9068 2023-04-22 14:55:20,701 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/d32d9ff37e124c8fa9929bc9304c86b9 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/d32d9ff37e124c8fa9929bc9304c86b9 2023-04-22 14:55:20,702 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/d81bb39725e547c5928e08322cd3d0fc to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/d81bb39725e547c5928e08322cd3d0fc 2023-04-22 14:55:20,704 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/e59d6f4dbd484aeb8af5f2bb230b76df to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/e59d6f4dbd484aeb8af5f2bb230b76df 2023-04-22 14:55:20,705 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/fce7ad84a178409790dfb08e9105e277 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf1/fce7ad84a178409790dfb08e9105e277 2023-04-22 14:55:20,708 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/2545531ea90443649dcde47cb1b95193 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/2545531ea90443649dcde47cb1b95193 2023-04-22 14:55:20,709 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/6c22ef2ae5d94307b5fa1806f2b8f524 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/6c22ef2ae5d94307b5fa1806f2b8f524 2023-04-22 14:55:20,710 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/722dd7f9abe94ee28f5eca217882c3cc to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/722dd7f9abe94ee28f5eca217882c3cc 2023-04-22 14:55:20,712 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8084ad5236144001a89606a4d13f3987 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8084ad5236144001a89606a4d13f3987 2023-04-22 14:55:20,713 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8a858a53393a45059de5bb28f6e60605 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/8a858a53393a45059de5bb28f6e60605 2023-04-22 14:55:20,714 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/bb1f0d3f411b4de8839f377bb9e7ac1b to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/bb1f0d3f411b4de8839f377bb9e7ac1b 2023-04-22 14:55:20,716 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/f153499e57544214aff18322c789e2d8 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/cf2/f153499e57544214aff18322c789e2d8 2023-04-22 14:55:20,718 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/recovered.edits/85.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164/recovered.edits/85.seqid 2023-04-22 14:55:20,719 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testInvalidSplitKey/99790a94004fb6fc0bfe1f00f245d164 2023-04-22 14:55:20,719 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testInvalidSplitKey regions 2023-04-22 14:55:20,722 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=108, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:20,723 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of testInvalidSplitKey from hbase:meta 2023-04-22 14:55:20,728 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testInvalidSplitKey' descriptor. 2023-04-22 14:55:20,729 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=108, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:20,729 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testInvalidSplitKey' from region states. 2023-04-22 14:55:20,730 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175320729"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:20,731 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-22 14:55:20,731 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 99790a94004fb6fc0bfe1f00f245d164, NAME => 'testInvalidSplitKey,,1682175318173.99790a94004fb6fc0bfe1f00f245d164.', STARTKEY => '', ENDKEY => ''}] 2023-04-22 14:55:20,731 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testInvalidSplitKey' as deleted. 2023-04-22 14:55:20,731 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testInvalidSplitKey","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175320731"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:20,733 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testInvalidSplitKey state from META 2023-04-22 14:55:20,734 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=108, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:20,736 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey in 57 msec 2023-04-22 14:55:20,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=108 2023-04-22 14:55:20,783 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testInvalidSplitKey, procId: 108 completed 2023-04-22 14:55:20,802 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testInvalidSplitKey Thread=369 (was 346) - Thread LEAK? -, OpenFileDescriptor=744 (was 655) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=343 (was 343), ProcessCount=171 (was 172), AvailableMemoryMB=12416 (was 12428) 2023-04-22 14:55:20,818 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=369, OpenFileDescriptor=744, MaxFileDescriptor=60000, SystemLoadAverage=343, ProcessCount=171, AvailableMemoryMB=12416 2023-04-22 14:55:20,818 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:55:20,820 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:55:20,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:55:20,822 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:20,824 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:55:20,825 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:55:20,827 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:20,828 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04 empty. 2023-04-22 14:55:20,828 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:20,828 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-04-22 14:55:20,841 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/.tabledesc/.tableinfo.0000000001 2023-04-22 14:55:20,843 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(7675): creating {ENCODED => db0da92be657e4731bf6c1fb5d561e04, NAME => 'testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testRecoveryAndDoubleExecution', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:55:20,859 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:20,859 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1604): Closing db0da92be657e4731bf6c1fb5d561e04, disabling compactions & flushes 2023-04-22 14:55:20,859 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:20,859 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:20,859 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. after waiting 0 ms 2023-04-22 14:55:20,859 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:20,859 INFO [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:20,860 DEBUG [RegionOpenAndInit-testRecoveryAndDoubleExecution-pool-0] regionserver.HRegion(1558): Region close journal for db0da92be657e4731bf6c1fb5d561e04: 2023-04-22 14:55:20,862 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:55:20,863 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175320863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175320863"}]},"ts":"1682175320863"} 2023-04-22 14:55:20,864 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:55:20,865 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:55:20,865 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175320865"}]},"ts":"1682175320865"} 2023-04-22 14:55:20,866 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLING in hbase:meta 2023-04-22 14:55:20,870 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:55:20,872 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:55:20,872 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:55:20,872 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:55:20,872 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:55:20,872 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, ASSIGN}] 2023-04-22 14:55:20,874 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, ASSIGN 2023-04-22 14:55:20,875 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=110, ppid=109, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:55:21,025 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:55:21,026 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=db0da92be657e4731bf6c1fb5d561e04, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,027 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175321026"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175321026"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175321026"}]},"ts":"1682175321026"} 2023-04-22 14:55:21,029 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=111, ppid=110, state=RUNNABLE; OpenRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:21,033 ERROR [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid 6cc12795-ed03-4ce7-985d-18d7c9a6ec2b) service to localhost/127.0.0.1:35843 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:21,190 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => db0da92be657e4731bf6c1fb5d561e04, NAME => 'testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:55:21,191 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:21,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:21,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:21,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:21,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:21,192 INFO [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:21,194 DEBUG [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1 2023-04-22 14:55:21,194 DEBUG [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1 2023-04-22 14:55:21,194 INFO [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region db0da92be657e4731bf6c1fb5d561e04 columnFamilyName cf1 2023-04-22 14:55:21,195 INFO [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] regionserver.HStore(310): Store=db0da92be657e4731bf6c1fb5d561e04/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:21,195 INFO [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:21,197 DEBUG [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2 2023-04-22 14:55:21,197 DEBUG [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2 2023-04-22 14:55:21,197 INFO [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region db0da92be657e4731bf6c1fb5d561e04 columnFamilyName cf2 2023-04-22 14:55:21,198 INFO [StoreOpener-db0da92be657e4731bf6c1fb5d561e04-1] regionserver.HStore(310): Store=db0da92be657e4731bf6c1fb5d561e04/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:21,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:21,199 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:21,201 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:21,202 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:21,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:55:21,205 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened db0da92be657e4731bf6c1fb5d561e04; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72848159, jitterRate=0.08552215993404388}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:21,205 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for db0da92be657e4731bf6c1fb5d561e04: 2023-04-22 14:55:21,206 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04., pid=111, masterSystemTime=1682175321187 2023-04-22 14:55:21,208 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,208 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,208 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=db0da92be657e4731bf6c1fb5d561e04, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,208 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175321208"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175321208"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175321208"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175321208"}]},"ts":"1682175321208"} 2023-04-22 14:55:21,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=111, resume processing ppid=110 2023-04-22 14:55:21,211 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=110, state=SUCCESS; OpenRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 in 182 msec 2023-04-22 14:55:21,213 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=110, resume processing ppid=109 2023-04-22 14:55:21,213 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=109, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, ASSIGN in 339 msec 2023-04-22 14:55:21,214 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:55:21,214 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175321214"}]},"ts":"1682175321214"} 2023-04-22 14:55:21,215 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=ENABLED in hbase:meta 2023-04-22 14:55:21,217 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=109, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRecoveryAndDoubleExecution execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:55:21,219 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution in 396 msec 2023-04-22 14:55:21,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:21,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-22 14:55:21,336 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:21,336 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,336 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-22 14:55:21,336 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:55:21,337 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,337 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,340 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,341 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,341 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,341 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,343 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,343 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,344 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,344 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,344 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,344 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,344 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,346 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,346 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,346 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,352 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:21,353 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,353 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. started... 2023-04-22 14:55:21,354 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing db0da92be657e4731bf6c1fb5d561e04 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-22 14:55:21,360 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,361 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,376 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/8efb0bdf832c43ed9291e22180632d45 2023-04-22 14:55:21,409 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/4dceddf0059949ab80c096bc5480a0f1 2023-04-22 14:55:21,418 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/8efb0bdf832c43ed9291e22180632d45 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45 2023-04-22 14:55:21,425 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:55:21,426 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/4dceddf0059949ab80c096bc5480a0f1 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1 2023-04-22 14:55:21,430 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:55:21,431 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for db0da92be657e4731bf6c1fb5d561e04 in 77ms, sequenceid=6, compaction requested=false 2023-04-22 14:55:21,431 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:21,431 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for db0da92be657e4731bf6c1fb5d561e04: 2023-04-22 14:55:21,431 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,432 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,434 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,434 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,434 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,434 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-22 14:55:21,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b14c37e[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:21,436 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:55:21,436 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,437 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,438 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,438 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:21,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,438 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,438 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:21,438 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:21,440 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:21,440 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:21,440 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,440 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,440 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:21,441 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:21,441 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,441 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,441 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,441 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:21,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:21,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:55:21,445 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-22 14:55:21,445 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:21,445 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:21,445 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,445 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:21,447 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,447 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,447 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,447 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,447 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,447 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,447 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,448 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,449 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,449 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,450 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,451 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,451 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,454 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:21,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:21,454 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,455 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,455 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,455 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:21,455 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,555 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:21,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:21,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:21,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:21,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-22 14:55:21,578 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:21,579 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,579 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-22 14:55:21,579 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:55:21,579 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,579 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,580 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,580 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,580 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,580 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,581 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,581 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:21,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,582 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,583 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,584 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:21,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,584 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,585 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,586 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,587 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,587 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:21,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,592 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:21,596 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:21,596 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,596 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. started... 2023-04-22 14:55:21,596 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing db0da92be657e4731bf6c1fb5d561e04 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:21,617 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/8550886b879649ca9fcfb3fd18ee90d3 2023-04-22 14:55:21,630 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b 2023-04-22 14:55:21,634 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/8550886b879649ca9fcfb3fd18ee90d3 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3 2023-04-22 14:55:21,638 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:55:21,639 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b 2023-04-22 14:55:21,644 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:55:21,645 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for db0da92be657e4731bf6c1fb5d561e04 in 49ms, sequenceid=19, compaction requested=false 2023-04-22 14:55:21,645 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for db0da92be657e4731bf6c1fb5d561e04: 2023-04-22 14:55:21,645 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,651 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,651 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-22 14:55:21,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dc1e3a2[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:21,654 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:55:21,654 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:21,655 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:21,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:21,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:55:21,661 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-22 14:55:21,661 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:21,661 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:21,661 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,661 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:21,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,664 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,664 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,664 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,665 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,666 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,667 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,667 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,668 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,670 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,670 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,671 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:21,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:21,671 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,672 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,672 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,672 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,672 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,672 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,672 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,672 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:21,673 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:21,773 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:21,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:21,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:21,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:21,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-22 14:55:21,788 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:21,788 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,788 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-22 14:55:21,788 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:55:21,789 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,789 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,790 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,790 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,790 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,790 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,790 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,790 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:21,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,792 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,793 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,793 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. started... 2023-04-22 14:55:21,793 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing db0da92be657e4731bf6c1fb5d561e04 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:21,794 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,810 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/1355a05336a34197a61fc8830ab1fee5 2023-04-22 14:55:21,827 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/67b568c023f543a380e0d32be460c7f6 2023-04-22 14:55:21,831 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/1355a05336a34197a61fc8830ab1fee5 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5 2023-04-22 14:55:21,835 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:55:21,836 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/67b568c023f543a380e0d32be460c7f6 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6 2023-04-22 14:55:21,840 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:55:21,841 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for db0da92be657e4731bf6c1fb5d561e04 in 48ms, sequenceid=32, compaction requested=true 2023-04-22 14:55:21,841 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for db0da92be657e4731bf6c1fb5d561e04: 2023-04-22 14:55:21,841 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,841 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:21,841 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:21,841 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,841 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,841 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,843 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-22 14:55:21,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2148c222[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:21,845 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:55:21,845 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,846 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,846 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:21,847 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:21,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,847 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:21,847 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:21,847 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:21,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:21,848 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:21,849 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:21,849 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,849 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:21,849 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,850 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,851 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:21,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:55:21,854 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-22 14:55:21,854 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:21,854 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:21,854 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,854 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:21,856 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,856 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:21,856 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,856 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,856 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,857 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,857 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,857 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,858 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:21,858 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:21,860 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:21,860 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,860 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,866 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,866 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,866 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,866 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,866 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,866 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,867 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:21,866 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:21,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,866 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,867 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:21,967 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:21,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:21,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:21,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:21,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-22 14:55:21,990 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:21,990 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,990 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-22 14:55:21,990 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:55:21,991 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,991 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,992 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:21,993 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,993 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:21,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,993 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,994 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:21,994 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:21,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:21,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:21,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,995 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:21,996 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:21,996 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:21,996 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:21,996 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. started... 2023-04-22 14:55:21,996 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing db0da92be657e4731bf6c1fb5d561e04 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:21,996 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,997 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,997 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:21,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:22,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:22,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:22,006 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,006 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,006 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:22,427 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/6239811eeb0944b5a53476450c9f4eb1 2023-04-22 14:55:22,440 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/3f5fdd5c527c4031a910d95996976f5b 2023-04-22 14:55:22,445 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/6239811eeb0944b5a53476450c9f4eb1 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1 2023-04-22 14:55:22,450 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:55:22,450 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/3f5fdd5c527c4031a910d95996976f5b as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b 2023-04-22 14:55:22,454 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:55:22,455 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for db0da92be657e4731bf6c1fb5d561e04 in 459ms, sequenceid=45, compaction requested=true 2023-04-22 14:55:22,456 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for db0da92be657e4731bf6c1fb5d561e04: 2023-04-22 14:55:22,456 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:22,456 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:22,458 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,458 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,458 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,458 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-22 14:55:22,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@160ee231[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:22,461 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:55:22,461 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:22,463 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:22,463 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:22,463 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:22,464 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,466 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:22,467 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,467 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,468 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:22,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:55:22,472 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-22 14:55:22,472 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:22,472 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:22,472 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,472 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:22,473 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,473 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,473 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,473 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,473 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,474 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,473 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,475 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,475 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,475 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,477 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,478 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,478 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,482 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,483 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,483 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:22,483 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:22,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,488 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,583 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:22,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:22,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:22,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-22 14:55:22,600 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:22,601 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,601 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-22 14:55:22,601 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:55:22,601 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,601 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,602 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,602 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,602 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:22,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:22,604 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,605 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:22,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,605 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:22,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:22,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:22,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:22,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:22,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:22,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:22,621 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:22,624 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,624 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,624 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:22,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:22,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:22,626 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:22,626 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:22,626 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. started... 2023-04-22 14:55:22,627 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing db0da92be657e4731bf6c1fb5d561e04 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:22,628 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,628 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,628 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:22,653 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/514568174e454d74ae5a0934056102f5 2023-04-22 14:55:22,686 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/09deadd706154e24a2ac1962040b099b 2023-04-22 14:55:22,696 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/514568174e454d74ae5a0934056102f5 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5 2023-04-22 14:55:22,700 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:55:22,701 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/09deadd706154e24a2ac1962040b099b as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b 2023-04-22 14:55:22,704 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:55:22,705 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for db0da92be657e4731bf6c1fb5d561e04 in 79ms, sequenceid=58, compaction requested=true 2023-04-22 14:55:22,705 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for db0da92be657e4731bf6c1fb5d561e04: 2023-04-22 14:55:22,705 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:22,706 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:22,706 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:22,706 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,706 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:22,706 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:22,713 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,713 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,713 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,714 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-22 14:55:22,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bbcccbd[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:22,715 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:55:22,715 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,716 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,716 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,716 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:22,717 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,717 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:22,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:22,717 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:22,718 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,718 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,719 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:22,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:22,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:55:22,722 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-22 14:55:22,722 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:22,722 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:22,722 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,722 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:22,725 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,725 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,725 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,726 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,727 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,728 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,729 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,729 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:22,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,732 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,733 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:22,733 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,732 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,733 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,733 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,733 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,733 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,733 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,733 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,833 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:22,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:22,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:22,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testRecoveryAndDoubleExecution was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:22,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testRecoveryAndDoubleExecution 2023-04-22 14:55:22,848 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:22,848 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,848 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testRecoveryAndDoubleExecution' starting 'acquire' 2023-04-22 14:55:22,848 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testRecoveryAndDoubleExecution', kicking off acquire phase on members. 2023-04-22 14:55:22,848 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,849 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,850 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,850 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,850 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,850 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,850 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,850 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:22,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:22,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:22,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:22,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:22,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:22,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:22,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testRecoveryAndDoubleExecution 2023-04-22 14:55:22,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,852 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testRecoveryAndDoubleExecution is still present, removing 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:22,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testRecoveryAndDoubleExecution' with timeout 60000ms 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testRecoveryAndDoubleExecution' starting 'acquire' stage 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:22,853 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:22,853 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:22,854 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. started... 2023-04-22 14:55:22,854 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing db0da92be657e4731bf6c1fb5d561e04 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:22,854 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,854 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:22,855 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:22,872 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/c20b295450e841beb99d4b9cc660d840 2023-04-22 14:55:22,905 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/25c06290379d4feab266db19a7466966 2023-04-22 14:55:22,913 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/c20b295450e841beb99d4b9cc660d840 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840 2023-04-22 14:55:22,917 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:55:22,918 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/25c06290379d4feab266db19a7466966 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966 2023-04-22 14:55:22,923 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:55:22,924 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for db0da92be657e4731bf6c1fb5d561e04 in 70ms, sequenceid=71, compaction requested=true 2023-04-22 14:55:22,924 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for db0da92be657e4731bf6c1fb5d561e04: 2023-04-22 14:55:22,924 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:22,924 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,925 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testRecoveryAndDoubleExecution' locally acquired 2023-04-22 14:55:22,925 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testRecoveryAndDoubleExecution) in zk 2023-04-22 14:55:22,927 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,927 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,928 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,928 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testRecoveryAndDoubleExecution' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testRecoveryAndDoubleExecution' on coordinator 2023-04-22 14:55:22,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1952e69b[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:22,930 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testRecoveryAndDoubleExecution' starting 'in-barrier' execution. 2023-04-22 14:55:22,930 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,931 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,931 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,931 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,931 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:22,931 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,931 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:22,931 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,931 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:22,931 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testRecoveryAndDoubleExecution' received 'reached' from coordinator. 2023-04-22 14:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testRecoveryAndDoubleExecution' locally completed 2023-04-22 14:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testRecoveryAndDoubleExecution' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:22,934 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:22,934 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,934 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:22,935 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:22,935 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,935 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:22,935 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,935 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testRecoveryAndDoubleExecution' has notified controller of completion 2023-04-22 14:55:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,935 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,935 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testRecoveryAndDoubleExecution' completed. 2023-04-22 14:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testRecoveryAndDoubleExecution' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:22,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testRecoveryAndDoubleExecution', counting down latch. Waiting for 0 more 2023-04-22 14:55:22,940 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testRecoveryAndDoubleExecution' execution completed 2023-04-22 14:55:22,940 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:22,940 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:22,940 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testRecoveryAndDoubleExecution 2023-04-22 14:55:22,970 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testRecoveryAndDoubleExecutionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:22,973 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,973 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,973 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:22,973 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,974 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,973 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,973 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:22,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,974 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,975 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,975 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:22,977 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testRecoveryAndDoubleExecution 2023-04-22 14:55:22,977 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,978 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,983 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,983 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,983 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:22,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,983 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,983 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,983 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testRecoveryAndDoubleExecution' 2023-04-22 14:55:22,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:22,983 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:22,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:22,983 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:22,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:22,983 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testRecoveryAndDoubleExecution'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testRecoveryAndDoubleExecution 2023-04-22 14:55:22,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:23,084 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:23,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:23,345 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: true 2023-04-22 14:55:23,346 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:23,347 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,348 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,348 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,348 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,348 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:23,349 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:23,349 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:23,484 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:55:23,597 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 0 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,597 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:23,597 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:23,597 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:23,597 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (992075877) connection to localhost/127.0.0.1:35843 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-9,5,PEWorkerGroup] Thread[HFileArchiver-10,5,PEWorkerGroup] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] 2023-04-22 14:55:23,598 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:23,598 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:23,598 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:23,598 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:23,599 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:23,600 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[] 2023-04-22 14:55:23,601 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:23,601 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:23,601 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:23,602 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:23,602 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:23,602 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:23,602 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:23,602 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:23,603 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:23,603 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:23,603 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:23,603 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:23,604 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:23,604 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:23,604 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:23,604 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:23,604 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:23,605 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:23,605 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:23,605 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:23,605 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:23,606 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:23,606 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:23,606 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:23,606 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:23,607 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:23,607 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:23,607 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:23,608 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:23,608 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:23,608 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:23,608 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:23,608 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:23,609 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:23,609 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,609 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:23,609 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,609 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,609 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,609 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-04-22 14:55:23,610 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:23,611 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:23,611 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175353611 2023-04-22 14:55:23,613 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,613 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:23,616 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:23,616 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:23,616 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:23,616 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175383616 2023-04-22 14:55:23,616 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175443616 2023-04-22 14:55:23,617 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:55:23,617 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:23,619 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:23,619 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,619 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:23,619 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:23,867 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 1 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,867 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:23,867 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:23,867 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:23,867 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:23,868 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:23,868 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:23,868 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:23,868 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:23,868 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:23,870 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0] 2023-04-22 14:55:23,870 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:23,871 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:23,871 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:23,871 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:23,871 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:23,872 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:23,872 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:23,872 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:23,872 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:23,873 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:23,873 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:23,873 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:23,873 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:23,873 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:23,874 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:23,874 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:23,874 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:23,874 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:23,875 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:23,875 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:23,875 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:23,875 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:23,875 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:23,875 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:23,876 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:23,876 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:23,876 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:23,876 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:23,877 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:23,877 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:23,877 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:23,877 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:23,878 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:23,878 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:23,878 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,878 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:23,878 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,878 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,879 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,879 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 10 msec 2023-04-22 14:55:23,885 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:23,886 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:23,886 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175353886 2023-04-22 14:55:23,887 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:23,888 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,891 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:23,891 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:23,891 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:23,891 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175383891 2023-04-22 14:55:23,891 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175443891 2023-04-22 14:55:23,891 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:55:23,891 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:23,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN}] 2023-04-22 14:55:23,894 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:23,894 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:23,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:23,894 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:24,033 ERROR [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid cd7c3aae-8b07-4809-9172-b4e8dd2fdabd) service to localhost/127.0.0.1:35843 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:24,141 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 2 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,142 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:24,142 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:24,142 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:24,142 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:24,142 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:24,142 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:24,142 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:24,142 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:24,143 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:24,144 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1] 2023-04-22 14:55:24,144 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:24,144 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:24,145 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:24,145 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:24,145 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:24,145 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:24,145 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:24,145 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:24,145 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:24,146 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:24,146 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:24,146 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:24,146 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:24,146 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:24,147 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:24,147 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:24,147 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:24,147 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:24,147 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:24,147 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:24,148 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:24,148 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:24,148 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:24,148 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:24,148 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:24,149 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:24,149 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:24,149 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:24,149 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:24,149 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:24,150 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:24,150 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:24,150 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:24,150 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:24,151 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,151 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:24,151 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,151 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,151 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,151 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 8 msec 2023-04-22 14:55:24,154 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:24,155 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:24,156 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175354156 2023-04-22 14:55:24,156 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:24,162 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CLOSE_PARENT_REGION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,162 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:24,162 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:24,163 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:24,163 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175384163 2023-04-22 14:55:24,163 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175444163 2023-04-22 14:55:24,163 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 7 msec 2023-04-22 14:55:24,163 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN}] 2023-04-22 14:55:24,163 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:24,168 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=113, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN 2023-04-22 14:55:24,169 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:24,169 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175324169"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175324169"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175324169"}]},"ts":"1682175324169"} 2023-04-22 14:55:24,170 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=113, state=RUNNABLE; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:24,322 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,322 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing db0da92be657e4731bf6c1fb5d561e04, disabling compactions & flushes 2023-04-22 14:55:24,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:24,323 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:24,323 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. after waiting 0 ms 2023-04-22 14:55:24,323 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:24,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing db0da92be657e4731bf6c1fb5d561e04 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-22 14:55:24,333 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/da6e8870efc74495a1061fb69971ff67 2023-04-22 14:55:24,354 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/f62cc82159674d0d938490683a3a8746 2023-04-22 14:55:24,358 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf1/da6e8870efc74495a1061fb69971ff67 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67 2023-04-22 14:55:24,362 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:55:24,363 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/.tmp/cf2/f62cc82159674d0d938490683a3a8746 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746 2023-04-22 14:55:24,367 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:55:24,368 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for db0da92be657e4731bf6c1fb5d561e04 in 45ms, sequenceid=82, compaction requested=true 2023-04-22 14:55:24,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-22 14:55:24,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:24,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. 2023-04-22 14:55:24,373 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for db0da92be657e4731bf6c1fb5d561e04: 2023-04-22 14:55:24,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,376 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=113 updating hbase:meta row=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSED 2023-04-22 14:55:24,377 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175324376"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175324376"}]},"ts":"1682175324376"} 2023-04-22 14:55:24,379 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=113 2023-04-22 14:55:24,379 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 in 208 msec 2023-04-22 14:55:24,381 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=112 2023-04-22 14:55:24,381 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN in 216 msec 2023-04-22 14:55:24,384 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:24,384 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,384 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:24,384 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:24,413 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 3 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,413 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:24,413 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:24,414 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:24,414 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:24,414 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:24,414 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:24,414 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:24,414 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:24,415 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:24,417 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2] 2023-04-22 14:55:24,417 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:55:24,417 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:55:24,417 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:24,418 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:24,418 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:24,418 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:24,418 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:24,419 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:24,419 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:24,419 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:24,419 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:24,419 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:24,420 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:24,420 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:24,420 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:24,420 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:24,421 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:24,421 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:24,421 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:24,421 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:24,421 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:24,422 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:24,422 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:24,422 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:24,422 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:24,422 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:24,423 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:24,423 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:24,423 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:24,425 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:24,425 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:24,425 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:24,425 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:24,427 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:24,427 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:24,427 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:24,428 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,428 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN 2023-04-22 14:55:24,428 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:24,428 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:24,428 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,428 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,428 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,429 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:24,429 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:24,429 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 13 msec 2023-04-22 14:55:24,431 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:24,431 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:24,432 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175354432 2023-04-22 14:55:24,432 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:24,434 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGIONS_CHECK_CLOSED_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:24,436 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:24,437 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:24,437 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:24,437 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175384437 2023-04-22 14:55:24,437 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175444437 2023-04-22 14:55:24,437 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-22 14:55:24,440 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:24,451 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=db0da92be657e4731bf6c1fb5d561e04, threads=14 2023-04-22 14:55:24,451 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,451 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,451 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,453 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,461 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,464 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,468 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,472 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,476 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,477 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,477 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,480 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,488 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,488 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,781 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,816 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,816 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,850 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,886 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,886 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,901 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,949 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,984 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,985 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:24,991 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,143 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,208 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,307 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,307 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region db0da92be657e4731bf6c1fb5d561e04 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-22 14:55:25,344 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:25,344 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:25,344 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:25,344 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:25,437 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 4 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:25,438 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:25,438 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:25,438 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:25,438 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:25,438 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:25,438 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:25,438 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:25,438 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:25,439 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:25,441 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7] 2023-04-22 14:55:25,441 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:55:25,441 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:55:25,441 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:25,443 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:25,443 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:25,443 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:25,443 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:25,443 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:25,444 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:25,444 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:25,444 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:25,444 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:25,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:25,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:25,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:25,445 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:25,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:25,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:25,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:25,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:25,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:25,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:25,446 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:25,447 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:25,447 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:25,447 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:25,447 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:25,447 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:25,448 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:25,448 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:25,448 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:25,448 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:25,449 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:25,449 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:25,449 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:25,449 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:25,449 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:25,449 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN 2023-04-22 14:55:25,450 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:25,450 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:25,450 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:25,450 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:25,450 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:25,450 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:25,450 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:25,450 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 11 msec 2023-04-22 14:55:25,451 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:25,452 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:25,458 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175355458 2023-04-22 14:55:25,459 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:25,460 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:25,462 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:25,463 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:25,463 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:25,463 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175385463 2023-04-22 14:55:25,463 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175445463 2023-04-22 14:55:25,463 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:55:25,478 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=112 splitting 14 storefiles, region=db0da92be657e4731bf6c1fb5d561e04, threads=14 2023-04-22 14:55:25,481 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,481 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,484 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,484 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,488 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,489 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,489 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,490 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,490 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,492 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,496 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,497 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,498 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,498 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=112 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,733 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,770 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,773 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,800 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,800 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,804 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,822 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,823 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,829 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:25,834 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:26,063 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:26,141 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:26,144 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:26,144 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=112 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3 for region: db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:26,144 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=112 split storefiles for region db0da92be657e4731bf6c1fb5d561e04 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-22 14:55:26,201 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:26,207 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:55:26,209 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:55:26,209 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:26,209 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,209 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:26,209 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:26,213 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 5 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,213 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:26,214 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:26,214 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:26,214 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:26,214 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:26,214 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:26,214 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:26,214 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:26,215 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:26,216 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8] 2023-04-22 14:55:26,216 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:55:26,216 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:55:26,217 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:26,217 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:26,217 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:26,217 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:26,218 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:26,218 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:26,218 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:26,218 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:26,218 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:26,219 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:26,219 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:26,219 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:26,219 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:26,219 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:26,220 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:26,221 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:26,221 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:26,221 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:26,221 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:26,221 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:26,222 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,222 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,222 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:26,222 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,222 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,223 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:26,223 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:26,223 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:26,223 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:26,223 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,224 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN 2023-04-22 14:55:26,224 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:26,224 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:26,224 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,224 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,224 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,224 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:26,224 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:26,225 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 9 msec 2023-04-22 14:55:26,225 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:26,226 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:26,227 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175356227 2023-04-22 14:55:26,229 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_WRITE_MAX_SEQUENCE_ID_FILE; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,229 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:26,238 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:26,238 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:26,238 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:26,239 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175386239 2023-04-22 14:55:26,239 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175446239 2023-04-22 14:55:26,239 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 9 msec 2023-04-22 14:55:26,241 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:26,244 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:26,244 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,244 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:26,244 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:26,489 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 6 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,489 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:26,489 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:26,489 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:26,490 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:26,490 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:26,490 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:26,490 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:26,490 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:26,491 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:26,492 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9] 2023-04-22 14:55:26,492 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:55:26,492 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:55:26,492 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:26,492 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:26,492 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:26,493 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:26,493 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:26,493 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:26,493 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:26,493 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:26,493 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:26,493 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:26,494 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:26,494 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:26,494 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:26,494 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:26,494 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:26,494 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:26,494 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:26,494 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:26,494 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:26,495 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:26,495 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:26,495 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:26,495 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:26,495 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:26,495 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:26,495 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,496 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,496 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:26,496 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,496 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,496 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:26,497 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:26,497 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:26,497 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:26,497 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,498 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN 2023-04-22 14:55:26,498 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:26,498 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:26,498 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,498 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,498 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,498 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:26,498 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:26,498 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 7 msec 2023-04-22 14:55:26,499 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:26,499 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:26,500 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175356500 2023-04-22 14:55:26,502 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_BEFORE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,502 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:26,504 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:26,504 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:26,504 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:26,504 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175386504 2023-04-22 14:55:26,505 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175446505 2023-04-22 14:55:26,505 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:55:26,505 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:26,507 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175326507"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1682175326507"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1682175326507"}]},"ts":"1682175326507"} 2023-04-22 14:55:26,507 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175326507"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175326507"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175326507"}]},"ts":"1682175326507"} 2023-04-22 14:55:26,507 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175326507"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175326507"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175326507"}]},"ts":"1682175326507"} 2023-04-22 14:55:26,509 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:26,509 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,509 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:26,509 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:26,755 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 7 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,755 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:26,755 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:26,755 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:26,756 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:26,756 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:26,756 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:26,756 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:26,756 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:26,757 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:26,758 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10] 2023-04-22 14:55:26,759 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:55:26,759 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:55:26,759 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:26,759 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:26,760 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:26,760 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:26,760 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:26,760 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:26,761 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:26,761 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:26,761 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:26,762 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:26,762 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:26,762 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:26,763 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:26,764 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:26,764 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:26,765 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:26,765 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:26,765 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:26,765 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:26,765 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:26,766 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:26,766 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:26,766 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:26,767 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:26,767 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:26,767 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,768 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,768 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:26,768 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,768 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:26,769 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:26,769 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:26,769 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:26,769 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:26,770 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,770 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN 2023-04-22 14:55:26,770 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:26,770 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:26,770 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,770 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,770 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,770 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:26,770 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:26,771 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 13 msec 2023-04-22 14:55:26,776 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:26,777 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:26,780 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175356780 2023-04-22 14:55:26,781 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_UPDATE_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,781 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:26,784 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:26,784 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:26,784 INFO [Listener at localhost/43403] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,36267,1682175218583, table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:26,784 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=c91bdbb39e9ace999285634ed6f596ba, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-22 14:55:26,785 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=e25bb5e88376202c9021b62f68bfcec0, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-22 14:55:26,785 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:26,785 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175386785 2023-04-22 14:55:26,785 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175446785 2023-04-22 14:55:26,785 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 3 msec 2023-04-22 14:55:26,786 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.","families":{"info":[{"qualifier":"regioninfo","vlen":64,"tag":[],"timestamp":"1682175326785"},{"qualifier":"splitA","vlen":66,"tag":[],"timestamp":"1682175326785"},{"qualifier":"splitB","vlen":66,"tag":[],"timestamp":"1682175326785"}]},"ts":"1682175326785"} 2023-04-22 14:55:26,786 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175326785"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175326785"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175326785"}]},"ts":"1682175326785"} 2023-04-22 14:55:26,786 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175326785"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175326785"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175326785"}]},"ts":"1682175326785"} 2023-04-22 14:55:26,788 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:26,790 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:26,790 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:26,790 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:26,791 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:27,035 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 8 exec state=pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,035 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:27,035 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:27,036 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:27,036 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:27,036 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:27,036 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:27,036 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:27,036 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:27,037 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:27,038 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11] 2023-04-22 14:55:27,038 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:55:27,038 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:55:27,038 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:27,038 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:27,039 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:27,039 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:27,039 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:27,039 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:27,039 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:27,040 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:27,040 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:27,040 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:27,040 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:27,040 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:27,041 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:27,041 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:27,041 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:27,041 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:27,042 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:27,042 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:27,042 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:27,042 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:27,042 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:27,043 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:27,043 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:27,043 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:27,043 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:27,043 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:27,043 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:27,044 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:27,044 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:27,044 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:27,044 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:27,044 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:27,044 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:27,045 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:27,045 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,045 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN 2023-04-22 14:55:27,045 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:27,045 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:27,045 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,046 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,046 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,046 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:27,046 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:27,046 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 9 msec 2023-04-22 14:55:27,047 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:27,047 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:27,048 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175357048 2023-04-22 14:55:27,049 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:27,049 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_PRE_OPERATION_AFTER_META; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,052 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:27,053 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:27,053 INFO [Listener at localhost/43403] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,36267,1682175218583, table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:27,053 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=c91bdbb39e9ace999285634ed6f596ba, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-22 14:55:27,053 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=e25bb5e88376202c9021b62f68bfcec0, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-22 14:55:27,053 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:27,053 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175387053 2023-04-22 14:55:27,053 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175447053 2023-04-22 14:55:27,054 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 4 msec 2023-04-22 14:55:27,054 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:27,056 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, ASSIGN}] 2023-04-22 14:55:27,056 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:27,056 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,056 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:27,056 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:27,069 ERROR [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid 0e9964a3-eb0e-4f0a-bd76-2c6599875fd5) service to localhost/127.0.0.1:35843 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:27,304 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 9 exec state=pid=112, state=WAITING:SPLIT_TABLE_REGION_POST_OPERATION, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,304 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:27,304 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:27,304 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:27,304 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:27,305 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:27,305 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:27,305 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:27,305 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:27,306 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:27,307 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12] 2023-04-22 14:55:27,307 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:55:27,307 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:55:27,307 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:27,308 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:27,308 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:27,308 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:27,309 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:27,309 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:27,309 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:27,309 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:27,309 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:27,309 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:27,310 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:27,310 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:27,310 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:27,310 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:27,310 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:27,311 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:27,311 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:27,311 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:27,311 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:27,311 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:27,311 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:27,311 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:27,312 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:27,312 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:27,312 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:27,312 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:27,312 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:27,313 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:27,313 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:27,313 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:27,313 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:27,313 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:27,314 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:27,314 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:27,314 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,314 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN 2023-04-22 14:55:27,314 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:27,314 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:27,315 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,315 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,315 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,315 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:27,315 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:27,315 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 9 msec 2023-04-22 14:55:27,316 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:27,316 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:27,317 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175357317 2023-04-22 14:55:27,318 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:27,321 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_OPEN_CHILD_REGIONS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,323 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:27,324 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:27,324 INFO [Listener at localhost/43403] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,36267,1682175218583, table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:27,324 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=c91bdbb39e9ace999285634ed6f596ba, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-22 14:55:27,324 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=e25bb5e88376202c9021b62f68bfcec0, regionState=CLOSED, lastHost=null, regionLocation=null, openSeqNum=1 2023-04-22 14:55:27,324 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:27,324 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175387324 2023-04-22 14:55:27,325 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175447325 2023-04-22 14:55:27,325 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 6 msec 2023-04-22 14:55:27,325 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, ASSIGN}, {pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, ASSIGN}] 2023-04-22 14:55:27,325 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:27,327 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, ASSIGN 2023-04-22 14:55:27,328 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=116, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:55:27,329 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, ASSIGN 2023-04-22 14:55:27,329 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=115, ppid=112, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:55:27,478 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-22 14:55:27,480 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=c91bdbb39e9ace999285634ed6f596ba, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:27,480 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175327480"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175327480"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175327480"}]},"ts":"1682175327480"} 2023-04-22 14:55:27,482 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=115, state=RUNNABLE; OpenRegionProcedure c91bdbb39e9ace999285634ed6f596ba, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:27,484 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=e25bb5e88376202c9021b62f68bfcec0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:27,484 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175327484"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175327484"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175327484"}]},"ts":"1682175327484"} 2023-04-22 14:55:27,485 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=118, ppid=116, state=RUNNABLE; OpenRegionProcedure e25bb5e88376202c9021b62f68bfcec0, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:27,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:27,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c91bdbb39e9ace999285634ed6f596ba, NAME => 'testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.', STARTKEY => '', ENDKEY => '41'} 2023-04-22 14:55:27,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:27,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:27,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:27,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:27,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:27,638 INFO [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:27,639 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1 2023-04-22 14:55:27,639 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1 2023-04-22 14:55:27,640 INFO [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c91bdbb39e9ace999285634ed6f596ba columnFamilyName cf1 2023-04-22 14:55:27,652 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5-bottom 2023-04-22 14:55:27,661 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5-bottom 2023-04-22 14:55:27,666 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1-bottom 2023-04-22 14:55:27,670 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3-bottom 2023-04-22 14:55:27,674 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45-bottom 2023-04-22 14:55:27,678 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840-bottom 2023-04-22 14:55:27,687 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67-bottom 2023-04-22 14:55:27,687 INFO [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(310): Store=c91bdbb39e9ace999285634ed6f596ba/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:27,687 INFO [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:27,688 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2 2023-04-22 14:55:27,688 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2 2023-04-22 14:55:27,689 INFO [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c91bdbb39e9ace999285634ed6f596ba columnFamilyName cf2 2023-04-22 14:55:27,707 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b-bottom 2023-04-22 14:55:27,711 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966-bottom 2023-04-22 14:55:27,716 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b-bottom 2023-04-22 14:55:27,721 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b-bottom 2023-04-22 14:55:27,725 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1-bottom 2023-04-22 14:55:27,731 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6-bottom 2023-04-22 14:55:27,735 DEBUG [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746-bottom 2023-04-22 14:55:27,736 INFO [StoreOpener-c91bdbb39e9ace999285634ed6f596ba-1] regionserver.HStore(310): Store=c91bdbb39e9ace999285634ed6f596ba/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:27,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:27,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:27,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:27,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:27,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c91bdbb39e9ace999285634ed6f596ba; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73482788, jitterRate=0.09497886896133423}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:27,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c91bdbb39e9ace999285634ed6f596ba: 2023-04-22 14:55:27,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba., pid=117, masterSystemTime=1682175327633 2023-04-22 14:55:27,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:27,743 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:27,743 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:27,743 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:27,743 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:27,743 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1912): c91bdbb39e9ace999285634ed6f596ba/cf2 is initiating minor compaction (all files) 2023-04-22 14:55:27,743 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:27,743 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c91bdbb39e9ace999285634ed6f596ba/cf2 in testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:27,743 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1912): c91bdbb39e9ace999285634ed6f596ba/cf1 is initiating minor compaction (all files) 2023-04-22 14:55:27,744 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c91bdbb39e9ace999285634ed6f596ba/cf1 in testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:27,744 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/.tmp, totalSize=34.9 K 2023-04-22 14:55:27,744 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/.tmp, totalSize=34.9 K 2023-04-22 14:55:27,745 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175321325 2023-04-22 14:55:27,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:27,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:27,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:27,745 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175321325 2023-04-22 14:55:27,745 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175321557 2023-04-22 14:55:27,745 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e25bb5e88376202c9021b62f68bfcec0, NAME => 'testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.', STARTKEY => '41', ENDKEY => ''} 2023-04-22 14:55:27,745 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=115 updating hbase:meta row=c91bdbb39e9ace999285634ed6f596ba, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:27,745 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175327745"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175327745"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175327745"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175327745"}]},"ts":"1682175327745"} 2023-04-22 14:55:27,745 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:27,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRecoveryAndDoubleExecution e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:27,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,746 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,746 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175321774 2023-04-22 14:55:27,746 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175321557 2023-04-22 14:55:27,747 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175321774 2023-04-22 14:55:27,747 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175321970 2023-04-22 14:55:27,747 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175321970 2023-04-22 14:55:27,748 INFO [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,749 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175322585 2023-04-22 14:55:27,749 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175322585 2023-04-22 14:55:27,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=115 2023-04-22 14:55:27,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=115, state=SUCCESS; OpenRegionProcedure c91bdbb39e9ace999285634ed6f596ba, server=jenkins-hbase4.apache.org,36267,1682175218583 in 265 msec 2023-04-22 14:55:27,750 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175322835 2023-04-22 14:55:27,751 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175322835 2023-04-22 14:55:27,751 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1 2023-04-22 14:55:27,751 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1 2023-04-22 14:55:27,751 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175323086 2023-04-22 14:55:27,751 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175323086 2023-04-22 14:55:27,751 INFO [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e25bb5e88376202c9021b62f68bfcec0 columnFamilyName cf1 2023-04-22 14:55:27,753 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, ASSIGN in 425 msec 2023-04-22 14:55:27,776 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] throttle.PressureAwareThroughputController(145): c91bdbb39e9ace999285634ed6f596ba#cf1#compaction#108 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:27,778 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] throttle.PressureAwareThroughputController(145): c91bdbb39e9ace999285634ed6f596ba#cf2#compaction#109 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:27,782 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5-top 2023-04-22 14:55:27,785 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5-top 2023-04-22 14:55:27,790 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1-top 2023-04-22 14:55:27,794 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3-top 2023-04-22 14:55:27,798 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45-top 2023-04-22 14:55:27,810 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840-top 2023-04-22 14:55:27,815 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67-top 2023-04-22 14:55:27,816 INFO [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(310): Store=e25bb5e88376202c9021b62f68bfcec0/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:27,816 INFO [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,817 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2 2023-04-22 14:55:27,817 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2 2023-04-22 14:55:27,818 INFO [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e25bb5e88376202c9021b62f68bfcec0 columnFamilyName cf2 2023-04-22 14:55:27,854 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b-top 2023-04-22 14:55:27,855 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/.tmp/cf2/a9f959b45ecd4833bc289c96638612a3 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/a9f959b45ecd4833bc289c96638612a3 2023-04-22 14:55:27,861 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c91bdbb39e9ace999285634ed6f596ba/cf2 of c91bdbb39e9ace999285634ed6f596ba into a9f959b45ecd4833bc289c96638612a3(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:27,861 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c91bdbb39e9ace999285634ed6f596ba: 2023-04-22 14:55:27,861 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba., storeName=c91bdbb39e9ace999285634ed6f596ba/cf2, priority=9, startTime=1682175327743; duration=0sec 2023-04-22 14:55:27,861 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:27,863 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966-top 2023-04-22 14:55:27,867 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b-top 2023-04-22 14:55:27,871 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b-top 2023-04-22 14:55:27,881 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1-top 2023-04-22 14:55:27,885 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6-top 2023-04-22 14:55:27,888 DEBUG [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746-top 2023-04-22 14:55:27,889 INFO [StoreOpener-e25bb5e88376202c9021b62f68bfcec0-1] regionserver.HStore(310): Store=e25bb5e88376202c9021b62f68bfcec0/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:27,890 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,891 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,892 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testRecoveryAndDoubleExecution descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:27,897 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,897 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e25bb5e88376202c9021b62f68bfcec0; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62162815, jitterRate=-0.073701873421669}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:27,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e25bb5e88376202c9021b62f68bfcec0: 2023-04-22 14:55:27,898 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0., pid=118, masterSystemTime=1682175327633 2023-04-22 14:55:27,898 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:27,900 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:27,900 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:27,900 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:27,900 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1912): e25bb5e88376202c9021b62f68bfcec0/cf1 is initiating minor compaction (all files) 2023-04-22 14:55:27,900 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e25bb5e88376202c9021b62f68bfcec0/cf1 in testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:27,901 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67-top] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/.tmp, totalSize=34.9 K 2023-04-22 14:55:27,901 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175321325 2023-04-22 14:55:27,902 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175321557 2023-04-22 14:55:27,902 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175321774 2023-04-22 14:55:27,902 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682175321970 2023-04-22 14:55:27,903 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682175322585 2023-04-22 14:55:27,903 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:27,903 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:27,903 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1682175322835 2023-04-22 14:55:27,903 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=116 updating hbase:meta row=e25bb5e88376202c9021b62f68bfcec0, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:27,903 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175327903"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175327903"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175327903"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175327903"}]},"ts":"1682175327903"} 2023-04-22 14:55:27,904 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175323086 2023-04-22 14:55:27,907 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=118, resume processing ppid=116 2023-04-22 14:55:27,907 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=118, ppid=116, state=SUCCESS; OpenRegionProcedure e25bb5e88376202c9021b62f68bfcec0, server=jenkins-hbase4.apache.org,36267,1682175218583 in 419 msec 2023-04-22 14:55:27,909 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=112 2023-04-22 14:55:27,909 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, ASSIGN in 582 msec 2023-04-22 14:55:27,909 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: false 2023-04-22 14:55:27,910 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1759): TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:27,910 INFO [PEWorker-1] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:27,910 WARN [PEWorker-1] procedure2.ProcedureExecutor$WorkerThread(1980): Worker terminating UNNATURALLY null java.lang.RuntimeException: TESTING: Kill BEFORE store update: pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.kill(ProcedureExecutor.java:1761) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1703) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1392) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:73) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1964) 2023-04-22 14:55:27,915 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] throttle.PressureAwareThroughputController(145): e25bb5e88376202c9021b62f68bfcec0#cf1#compaction#110 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:27,935 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/.tmp/cf1/75e9e394e27340598f8cb209b576aa73 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/75e9e394e27340598f8cb209b576aa73 2023-04-22 14:55:27,940 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in e25bb5e88376202c9021b62f68bfcec0/cf1 of e25bb5e88376202c9021b62f68bfcec0 into 75e9e394e27340598f8cb209b576aa73(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:27,940 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e25bb5e88376202c9021b62f68bfcec0: 2023-04-22 14:55:27,940 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0., storeName=e25bb5e88376202c9021b62f68bfcec0/cf1, priority=9, startTime=1682175327898; duration=0sec 2023-04-22 14:55:27,940 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:27,940 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:27,941 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:27,941 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1912): e25bb5e88376202c9021b62f68bfcec0/cf2 is initiating minor compaction (all files) 2023-04-22 14:55:27,941 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e25bb5e88376202c9021b62f68bfcec0/cf2 in testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:27,941 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746-top] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/.tmp, totalSize=34.9 K 2023-04-22 14:55:27,942 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175321325 2023-04-22 14:55:27,942 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175321557 2023-04-22 14:55:27,943 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175321774 2023-04-22 14:55:27,943 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682175321970 2023-04-22 14:55:27,943 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682175322585 2023-04-22 14:55:27,944 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1682175322835 2023-04-22 14:55:27,944 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175323086 2023-04-22 14:55:27,967 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] throttle.PressureAwareThroughputController(145): e25bb5e88376202c9021b62f68bfcec0#cf2#compaction#111 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:27,990 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/.tmp/cf2/93085e2784574848adf7494dbbe63e79 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/93085e2784574848adf7494dbbe63e79 2023-04-22 14:55:27,994 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in e25bb5e88376202c9021b62f68bfcec0/cf2 of e25bb5e88376202c9021b62f68bfcec0 into 93085e2784574848adf7494dbbe63e79(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:27,994 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e25bb5e88376202c9021b62f68bfcec0: 2023-04-22 14:55:27,994 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0., storeName=e25bb5e88376202c9021b62f68bfcec0/cf2, priority=9, startTime=1682175327900; duration=0sec 2023-04-22 14:55:27,994 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:28,075 INFO [Listener at localhost/43403] procedure.MasterProcedureTestingUtility(454): Restart 10 exec state=pid=112, state=SUCCESS, locked=true; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,075 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(118): RESTART - Stop 2023-04-22 14:55:28,075 INFO [Listener at localhost/43403] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:28,075 INFO [Listener at localhost/43403] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:28,076 ERROR [Listener at localhost/43403] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] 2023-04-22 14:55:28,076 INFO [Listener at localhost/43403] procedure2.ProcedureTestingUtility(130): RESTART - Start 2023-04-22 14:55:28,076 INFO [Listener at localhost/43403] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=1 2023-04-22 14:55:28,076 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(562): Starting 1 core workers (bigger of cpus/4 or 16) with max (burst) worker count=10 2023-04-22 14:55:28,076 INFO [Listener at localhost/43403] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-22 14:55:28,077 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 0 msec 2023-04-22 14:55:28,078 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=112, ppid=-1, class=org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure) stack ids=[0, 1, 2, 7, 8, 9, 10, 11, 12, 13] 2023-04-22 14:55:28,078 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=113, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[3, 6] 2023-04-22 14:55:28,079 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=114, ppid=113, class=org.apache.hadoop.hbase.master.assignment.CloseRegionProcedure) stack ids=[4, 5] 2023-04-22 14:55:28,079 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=115, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[15, 16, 21] 2023-04-22 14:55:28,079 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=117, ppid=115, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[17, 20] 2023-04-22 14:55:28,079 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=116, ppid=112, class=org.apache.hadoop.hbase.master.assignment.TransitRegionStateProcedure) stack ids=[14, 18, 23] 2023-04-22 14:55:28,079 DEBUG [Listener at localhost/43403] store.ProcedureTree(114): Procedure Procedure(pid=118, ppid=116, class=org.apache.hadoop.hbase.master.assignment.OpenRegionProcedure) stack ids=[19, 22] 2023-04-22 14:55:28,079 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta 2023-04-22 14:55:28,080 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace 2023-04-22 14:55:28,080 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default 2023-04-22 14:55:28,080 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase 2023-04-22 14:55:28,080 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=9, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:28,080 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=12, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionUnevenDaughter, parent=1539dac144b8acabc3b8d8b035e2f099, daughterA=ce3321b32c9468788bb2b99657f8761a, daughterB=cbd04a56b211199f6d2f4a000c3bb4f6 2023-04-22 14:55:28,081 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=19, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:28,081 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=24, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionUnevenDaughter 2023-04-22 14:55:28,081 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=25, state=SUCCESS; CreateTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:28,081 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=28, state=SUCCESS; SplitTableRegionProcedure table=testSplitWithoutPONR, parent=b8acaab15ee577f218a114962304cb84, daughterA=a8630043203581c85952b1c770f58cd9, daughterB=bcf49cb18a7738ad1232f3887a8bd9af 2023-04-22 14:55:28,081 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=35, state=SUCCESS; DisableTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:28,082 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=40, state=SUCCESS; DeleteTableProcedure table=testSplitWithoutPONR 2023-04-22 14:55:28,082 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=41, state=SUCCESS; CreateTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:28,082 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=44, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.procedure2.ProcedureAbortedException via SplitTableRegionProcedure:org.apache.hadoop.hbase.procedure2.ProcedureAbortedException: abort requested; SplitTableRegionProcedure table=testRollbackAndDoubleExecution, parent=68494e94cc440349573fb09884f7f6bc, daughterA=4d7e9272d5568801647adcb548735695, daughterB=36622fde6be694d7ba2dd620c7da97ed 2023-04-22 14:55:28,082 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=47, state=SUCCESS; TransitRegionStateProcedure table=testRollbackAndDoubleExecution, region=68494e94cc440349573fb09884f7f6bc, ASSIGN 2023-04-22 14:55:28,082 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=49, state=SUCCESS; DisableTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:28,083 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=52, state=SUCCESS; DeleteTableProcedure table=testRollbackAndDoubleExecution 2023-04-22 14:55:28,083 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=53, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:28,083 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=56, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionNoStoreFile, parent=efad26bc745f89498b20ab3139f9002e, daughterA=3773e0571cc154fea12efd9885f8a72e, daughterB=4d87b2e1a535264e1cee0f8f8d63d62e 2023-04-22 14:55:28,083 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=63, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:28,083 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=68, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionNoStoreFile 2023-04-22 14:55:28,084 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=69, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:28,084 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=72, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionDeletedRowsDaughter, parent=001cf566a5015e8ef1438c27e2e9c824, daughterA=d6c36ebab54e40f448712ec2278adb5d, daughterB=302b059e322b8518325c4f182c8edf68 2023-04-22 14:55:28,084 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=79, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:28,084 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=84, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionDeletedRowsDaughter 2023-04-22 14:55:28,084 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=85, state=SUCCESS; CreateTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:28,085 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=88, state=SUCCESS; ModifyTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:28,085 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=95, state=ROLLEDBACK, exception=org.apache.hadoop.hbase.HBaseIOException via master-split-regions:org.apache.hadoop.hbase.HBaseIOException: The parent region state=OPENING, location=jenkins-hbase4.apache.org,33597,1682175218631, table=testRollbackForSplitTableRegionWithReplica, region=436aea79dba9499e96e168de6ff0b85b is currently in transition, give up; SplitTableRegionProcedure table=testRollbackForSplitTableRegionWithReplica, parent=be1c77199f735729a41e26e3a9a83c36, daughterA=b693833d4d2906e7e08ad076560cb6c2, daughterB=64f91c84e3512fb1c8db065f19815d67 2023-04-22 14:55:28,085 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=96, state=SUCCESS; DisableTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:28,085 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=101, state=SUCCESS; DeleteTableProcedure table=testRollbackForSplitTableRegionWithReplica 2023-04-22 14:55:28,085 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=102, state=SUCCESS; CreateTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:28,085 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=105, state=SUCCESS; DisableTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:28,086 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=108, state=SUCCESS; DeleteTableProcedure table=testInvalidSplitKey 2023-04-22 14:55:28,086 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(411): Completed pid=109, state=SUCCESS; CreateTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:28,087 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,087 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN 2023-04-22 14:55:28,087 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:28,087 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, ASSIGN 2023-04-22 14:55:28,088 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, ASSIGN 2023-04-22 14:55:28,088 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=117, ppid=115, state=SUCCESS; OpenRegionProcedure c91bdbb39e9ace999285634ed6f596ba, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:28,088 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(468): Loading pid=118, ppid=116, state=SUCCESS; OpenRegionProcedure e25bb5e88376202c9021b62f68bfcec0, server=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:28,088 DEBUG [Listener at localhost/43403] procedure2.Procedure(966): pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 held the lock before restarting, call acquireLock to restore it. 2023-04-22 14:55:28,088 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,088 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,088 INFO [Listener at localhost/43403] procedure.MasterProcedureScheduler(727): Took xlock for pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,088 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=113, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04, UNASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:28,088 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=114, ppid=113, state=SUCCESS; CloseRegionProcedure db0da92be657e4731bf6c1fb5d561e04, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:28,089 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=115, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:28,089 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=116, ppid=112, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, ASSIGN didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:28,089 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=117, ppid=115, state=SUCCESS; OpenRegionProcedure c91bdbb39e9ace999285634ed6f596ba, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:28,089 DEBUG [Listener at localhost/43403] procedure2.Procedure(946): pid=118, ppid=116, state=SUCCESS; OpenRegionProcedure e25bb5e88376202c9021b62f68bfcec0, server=jenkins-hbase4.apache.org,36267,1682175218583 didn't hold the lock before restarting, skip acquiring lock. 2023-04-22 14:55:28,089 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 11 msec 2023-04-22 14:55:28,092 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(253): hbase:meta replica znodes: [meta-region-server] 2023-04-22 14:55:28,093 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(272): Loaded hbase:meta state=OPEN, location=jenkins-hbase4.apache.org,35417,1682175218529, table=hbase:meta, region=1588230740 2023-04-22 14:55:28,096 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1682175358096 2023-04-22 14:55:28,099 DEBUG [PEWorker-1] procedure2.ProcedureExecutor(1400): LOCK_EVENT_WAIT pid=112, state=RUNNABLE:SPLIT_TABLE_REGION_POST_OPERATION; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,100 DEBUG [Listener at localhost/43403] assignment.AssignmentManager(1517): Joining cluster... 2023-04-22 14:55:28,104 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=d65d90b6fbc4a32d4609259a87039a65, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:28,104 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=db0da92be657e4731bf6c1fb5d561e04, regionState=CLOSED, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=2 2023-04-22 14:55:28,104 INFO [Listener at localhost/43403] assignment.RegionStates(657): Added to offline, CURRENTLY NEVER CLEARED!!! state=CLOSED, location=jenkins-hbase4.apache.org,36267,1682175218583, table=testRecoveryAndDoubleExecution, region=db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,105 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=c91bdbb39e9ace999285634ed6f596ba, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=86 2023-04-22 14:55:28,105 DEBUG [Listener at localhost/43403] assignment.RegionStateStore(147): Load hbase:meta entry region=e25bb5e88376202c9021b62f68bfcec0, regionState=OPEN, lastHost=jenkins-hbase4.apache.org,36267,1682175218583, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583, openSeqNum=86 2023-04-22 14:55:28,105 INFO [Listener at localhost/43403] assignment.AssignmentManager(1529): Number of RegionServers=3 2023-04-22 14:55:28,105 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1682175388105 2023-04-22 14:55:28,105 INFO [Listener at localhost/43403] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1682175448105 2023-04-22 14:55:28,105 INFO [Listener at localhost/43403] assignment.AssignmentManager(1536): Joined the cluster in 5 msec 2023-04-22 14:55:28,105 WARN [PEWorker-1] procedure2.ProcedureExecutor$Testing(122): Toggle KILL before store update to: true 2023-04-22 14:55:28,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=112, state=SUCCESS; SplitTableRegionProcedure table=testRecoveryAndDoubleExecution, parent=db0da92be657e4731bf6c1fb5d561e04, daughterA=c91bdbb39e9ace999285634ed6f596ba, daughterB=e25bb5e88376202c9021b62f68bfcec0 in 4.7590 sec 2023-04-22 14:55:28,246 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/.tmp/cf1/4f10173952e44790ab61fc5ef0b18685 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/4f10173952e44790ab61fc5ef0b18685 2023-04-22 14:55:28,252 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in c91bdbb39e9ace999285634ed6f596ba/cf1 of c91bdbb39e9ace999285634ed6f596ba into 4f10173952e44790ab61fc5ef0b18685(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:28,252 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c91bdbb39e9ace999285634ed6f596ba: 2023-04-22 14:55:28,252 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba., storeName=c91bdbb39e9ace999285634ed6f596ba/cf1, priority=9, startTime=1682175327742; duration=0sec 2023-04-22 14:55:28,252 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:28,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36267] ipc.CallRunner(144): callId: 860 service: ClientService methodName: Scan size: 154 connection: 172.31.14.131:50214 deadline: 1682175388356, exception=org.apache.hadoop.hbase.NotServingRegionException: testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04. is not online on jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:28,471 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-22 14:55:28,477 INFO [Listener at localhost/43403] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-04-22 14:55:28,479 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-22 14:55:28,495 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:55:28,496 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testRecoveryAndDoubleExecution 2023-04-22 14:55:28,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testRecoveryAndDoubleExecution 2023-04-22 14:55:28,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:28,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-04-22 14:55:28,501 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175328501"}]},"ts":"1682175328501"} 2023-04-22 14:55:28,502 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLING in hbase:meta 2023-04-22 14:55:28,505 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRecoveryAndDoubleExecution to state=DISABLING 2023-04-22 14:55:28,505 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, UNASSIGN}, {pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, UNASSIGN}] 2023-04-22 14:55:28,506 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, UNASSIGN 2023-04-22 14:55:28,507 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=e25bb5e88376202c9021b62f68bfcec0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:28,507 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175328507"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175328507"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175328507"}]},"ts":"1682175328507"} 2023-04-22 14:55:28,508 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=122, ppid=121, state=RUNNABLE; CloseRegionProcedure e25bb5e88376202c9021b62f68bfcec0, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:28,510 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=120, ppid=119, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, UNASSIGN 2023-04-22 14:55:28,511 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=c91bdbb39e9ace999285634ed6f596ba, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:28,511 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175328511"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175328511"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175328511"}]},"ts":"1682175328511"} 2023-04-22 14:55:28,512 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=123, ppid=120, state=RUNNABLE; CloseRegionProcedure c91bdbb39e9ace999285634ed6f596ba, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:28,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-04-22 14:55:28,660 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e25bb5e88376202c9021b62f68bfcec0, disabling compactions & flushes 2023-04-22 14:55:28,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:28,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:28,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. after waiting 0 ms 2023-04-22 14:55:28,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:28,661 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67-top] to archive 2023-04-22 14:55:28,662 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:28,664 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,665 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,667 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,668 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,669 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,671 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,672 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,674 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746-top] to archive 2023-04-22 14:55:28,674 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:28,676 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,678 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,679 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,680 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,681 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,683 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,684 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:55:28,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:28,688 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0. 2023-04-22 14:55:28,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e25bb5e88376202c9021b62f68bfcec0: 2023-04-22 14:55:28,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:28,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c91bdbb39e9ace999285634ed6f596ba, disabling compactions & flushes 2023-04-22 14:55:28,691 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:28,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:28,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. after waiting 0 ms 2023-04-22 14:55:28,691 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:28,691 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=e25bb5e88376202c9021b62f68bfcec0, regionState=CLOSED 2023-04-22 14:55:28,691 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175328691"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175328691"}]},"ts":"1682175328691"} 2023-04-22 14:55:28,693 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67-bottom] to archive 2023-04-22 14:55:28,694 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:28,696 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=122, resume processing ppid=121 2023-04-22 14:55:28,696 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=121, state=SUCCESS; CloseRegionProcedure e25bb5e88376202c9021b62f68bfcec0, server=jenkins-hbase4.apache.org,36267,1682175218583 in 185 msec 2023-04-22 14:55:28,697 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,698 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=e25bb5e88376202c9021b62f68bfcec0, UNASSIGN in 191 msec 2023-04-22 14:55:28,699 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,700 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,701 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,702 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,703 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,705 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,708 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746-bottom] to archive 2023-04-22 14:55:28,708 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:28,711 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,712 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,713 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,714 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,715 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,716 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,717 DEBUG [StoreCloser-testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:55:28,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:28,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba. 2023-04-22 14:55:28,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c91bdbb39e9ace999285634ed6f596ba: 2023-04-22 14:55:28,724 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:28,725 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=120 updating hbase:meta row=c91bdbb39e9ace999285634ed6f596ba, regionState=CLOSED 2023-04-22 14:55:28,725 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.","families":{"info":[{"qualifier":"regioninfo","vlen":66,"tag":[],"timestamp":"1682175328725"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175328725"}]},"ts":"1682175328725"} 2023-04-22 14:55:28,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=123, resume processing ppid=120 2023-04-22 14:55:28,733 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; CloseRegionProcedure c91bdbb39e9ace999285634ed6f596ba, server=jenkins-hbase4.apache.org,36267,1682175218583 in 214 msec 2023-04-22 14:55:28,735 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=120, resume processing ppid=119 2023-04-22 14:55:28,735 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=120, ppid=119, state=SUCCESS; TransitRegionStateProcedure table=testRecoveryAndDoubleExecution, region=c91bdbb39e9ace999285634ed6f596ba, UNASSIGN in 228 msec 2023-04-22 14:55:28,736 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175328736"}]},"ts":"1682175328736"} 2023-04-22 14:55:28,737 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRecoveryAndDoubleExecution, state=DISABLED in hbase:meta 2023-04-22 14:55:28,738 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRecoveryAndDoubleExecution to state=DISABLED 2023-04-22 14:55:28,741 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; DisableTableProcedure table=testRecoveryAndDoubleExecution in 243 msec 2023-04-22 14:55:28,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-04-22 14:55:28,803 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRecoveryAndDoubleExecution, procId: 119 completed 2023-04-22 14:55:28,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testRecoveryAndDoubleExecution 2023-04-22 14:55:28,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:28,806 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=124, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:28,807 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=124, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:28,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=124 2023-04-22 14:55:28,811 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,812 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:28,812 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,814 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/recovered.edits] 2023-04-22 14:55:28,814 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/recovered.edits] 2023-04-22 14:55:28,814 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/recovered.edits] 2023-04-22 14:55:28,823 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/.regioninfo to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/.regioninfo 2023-04-22 14:55:28,823 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/75e9e394e27340598f8cb209b576aa73 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf1/75e9e394e27340598f8cb209b576aa73 2023-04-22 14:55:28,824 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/1355a05336a34197a61fc8830ab1fee5 2023-04-22 14:55:28,826 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/514568174e454d74ae5a0934056102f5 2023-04-22 14:55:28,827 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/93085e2784574848adf7494dbbe63e79 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/cf2/93085e2784574848adf7494dbbe63e79 2023-04-22 14:55:28,828 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,828 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/6239811eeb0944b5a53476450c9f4eb1 2023-04-22 14:55:28,829 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,830 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8550886b879649ca9fcfb3fd18ee90d3 2023-04-22 14:55:28,831 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/.regioninfo to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/.regioninfo 2023-04-22 14:55:28,831 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,831 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/8efb0bdf832c43ed9291e22180632d45 2023-04-22 14:55:28,834 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,834 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/c20b295450e841beb99d4b9cc660d840 2023-04-22 14:55:28,836 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,836 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf1/da6e8870efc74495a1061fb69971ff67 2023-04-22 14:55:28,837 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/1355a05336a34197a61fc8830ab1fee5.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,839 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,841 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/514568174e454d74ae5a0934056102f5.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,842 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,843 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/09deadd706154e24a2ac1962040b099b 2023-04-22 14:55:28,843 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/6239811eeb0944b5a53476450c9f4eb1.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,845 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/8550886b879649ca9fcfb3fd18ee90d3.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,847 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25c06290379d4feab266db19a7466966 2023-04-22 14:55:28,847 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/8efb0bdf832c43ed9291e22180632d45.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,848 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,851 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/c20b295450e841beb99d4b9cc660d840.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,854 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf1/da6e8870efc74495a1061fb69971ff67.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,856 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,857 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,859 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,861 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,864 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,865 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/09deadd706154e24a2ac1962040b099b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,866 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b 2023-04-22 14:55:28,867 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/25c06290379d4feab266db19a7466966.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,868 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/c91bdbb39e9ace999285634ed6f596ba/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,869 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/25f7cb7a1fbe4ebeb37a02eaa9d6502b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,869 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/3f5fdd5c527c4031a910d95996976f5b 2023-04-22 14:55:28,871 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/3f5fdd5c527c4031a910d95996976f5b.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,871 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/4dceddf0059949ab80c096bc5480a0f1 2023-04-22 14:55:28,872 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/4f10173952e44790ab61fc5ef0b18685 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf1/4f10173952e44790ab61fc5ef0b18685 2023-04-22 14:55:28,873 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/4dceddf0059949ab80c096bc5480a0f1.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,874 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/67b568c023f543a380e0d32be460c7f6 2023-04-22 14:55:28,876 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/a9f959b45ecd4833bc289c96638612a3 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/cf2/a9f959b45ecd4833bc289c96638612a3 2023-04-22 14:55:28,876 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/cf2/f62cc82159674d0d938490683a3a8746 2023-04-22 14:55:28,877 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/67b568c023f543a380e0d32be460c7f6.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,878 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/e25bb5e88376202c9021b62f68bfcec0/cf2/f62cc82159674d0d938490683a3a8746.db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,880 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/recovered.edits/85.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04/recovered.edits/85.seqid 2023-04-22 14:55:28,882 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/db0da92be657e4731bf6c1fb5d561e04 2023-04-22 14:55:28,883 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba/recovered.edits/90.seqid 2023-04-22 14:55:28,883 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/c91bdbb39e9ace999285634ed6f596ba 2023-04-22 14:55:28,884 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0/recovered.edits/90.seqid 2023-04-22 14:55:28,884 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testRecoveryAndDoubleExecution/e25bb5e88376202c9021b62f68bfcec0 2023-04-22 14:55:28,885 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testRecoveryAndDoubleExecution regions 2023-04-22 14:55:28,887 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=124, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:28,889 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testRecoveryAndDoubleExecution from hbase:meta 2023-04-22 14:55:28,891 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testRecoveryAndDoubleExecution' descriptor. 2023-04-22 14:55:28,892 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=124, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:28,892 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testRecoveryAndDoubleExecution' from region states. 2023-04-22 14:55:28,892 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175328892"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:28,893 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175328892"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:28,893 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175328892"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:28,896 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-22 14:55:28,896 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => db0da92be657e4731bf6c1fb5d561e04, NAME => 'testRecoveryAndDoubleExecution,,1682175320821.db0da92be657e4731bf6c1fb5d561e04.', STARTKEY => '', ENDKEY => '', OFFLINE => true, SPLIT => true}, {ENCODED => c91bdbb39e9ace999285634ed6f596ba, NAME => 'testRecoveryAndDoubleExecution,,1682175323346.c91bdbb39e9ace999285634ed6f596ba.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => e25bb5e88376202c9021b62f68bfcec0, NAME => 'testRecoveryAndDoubleExecution,41,1682175323346.e25bb5e88376202c9021b62f68bfcec0.', STARTKEY => '41', ENDKEY => ''}] 2023-04-22 14:55:28,896 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testRecoveryAndDoubleExecution' as deleted. 2023-04-22 14:55:28,897 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRecoveryAndDoubleExecution","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175328896"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:28,898 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testRecoveryAndDoubleExecution state from META 2023-04-22 14:55:28,902 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=124, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRecoveryAndDoubleExecution 2023-04-22 14:55:28,907 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, state=SUCCESS; DeleteTableProcedure table=testRecoveryAndDoubleExecution in 98 msec 2023-04-22 14:55:28,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=124 2023-04-22 14:55:28,908 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRecoveryAndDoubleExecution, procId: 124 completed 2023-04-22 14:55:28,929 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testRecoveryAndDoubleExecution Thread=374 (was 369) - Thread LEAK? -, OpenFileDescriptor=833 (was 744) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=337 (was 343), ProcessCount=170 (was 171), AvailableMemoryMB=12365 (was 12416) 2023-04-22 14:55:28,946 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=374, OpenFileDescriptor=833, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=170, AvailableMemoryMB=12365 2023-04-22 14:55:28,946 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:55:28,948 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:55:28,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:55:28,949 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-22 14:55:28,951 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:55:28,952 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:55:28,953 DEBUG [HFileArchiver-16] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:28,954 DEBUG [HFileArchiver-16] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34 empty. 2023-04-22 14:55:28,954 DEBUG [HFileArchiver-16] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:28,954 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-04-22 14:55:28,966 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/.tabledesc/.tableinfo.0000000001 2023-04-22 14:55:28,976 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6ec5b9b2544c83568def9f0f3c2dac34, NAME => 'testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegionEmptyDaughter', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:55:28,987 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:28,988 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1604): Closing 6ec5b9b2544c83568def9f0f3c2dac34, disabling compactions & flushes 2023-04-22 14:55:28,988 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:28,988 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:28,988 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. after waiting 0 ms 2023-04-22 14:55:28,988 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:28,988 INFO [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:28,988 DEBUG [RegionOpenAndInit-testSplitTableRegionEmptyDaughter-pool-0] regionserver.HRegion(1558): Region close journal for 6ec5b9b2544c83568def9f0f3c2dac34: 2023-04-22 14:55:28,990 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:55:28,991 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175328991"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175328991"}]},"ts":"1682175328991"} 2023-04-22 14:55:28,992 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:55:28,993 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:55:28,993 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175328993"}]},"ts":"1682175328993"} 2023-04-22 14:55:28,994 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLING in hbase:meta 2023-04-22 14:55:28,999 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:55:29,000 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:55:29,000 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:55:29,000 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:55:29,000 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:55:29,000 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=6ec5b9b2544c83568def9f0f3c2dac34, ASSIGN}] 2023-04-22 14:55:29,002 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=6ec5b9b2544c83568def9f0f3c2dac34, ASSIGN 2023-04-22 14:55:29,002 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=125, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=6ec5b9b2544c83568def9f0f3c2dac34, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,33597,1682175218631; forceNewPlan=false, retain=false 2023-04-22 14:55:29,034 ERROR [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid 6cc12795-ed03-4ce7-985d-18d7c9a6ec2b) service to localhost/127.0.0.1:35843 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:29,152 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:55:29,154 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=6ec5b9b2544c83568def9f0f3c2dac34, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,154 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175329154"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175329154"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175329154"}]},"ts":"1682175329154"} 2023-04-22 14:55:29,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure 6ec5b9b2544c83568def9f0f3c2dac34, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:55:29,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:29,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6ec5b9b2544c83568def9f0f3c2dac34, NAME => 'testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:55:29,316 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:29,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:29,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:29,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:29,317 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:29,328 INFO [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:29,330 DEBUG [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1 2023-04-22 14:55:29,330 DEBUG [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1 2023-04-22 14:55:29,331 INFO [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6ec5b9b2544c83568def9f0f3c2dac34 columnFamilyName cf1 2023-04-22 14:55:29,331 INFO [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] regionserver.HStore(310): Store=6ec5b9b2544c83568def9f0f3c2dac34/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:29,331 INFO [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:29,333 DEBUG [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2 2023-04-22 14:55:29,333 DEBUG [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2 2023-04-22 14:55:29,333 INFO [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6ec5b9b2544c83568def9f0f3c2dac34 columnFamilyName cf2 2023-04-22 14:55:29,334 INFO [StoreOpener-6ec5b9b2544c83568def9f0f3c2dac34-1] regionserver.HStore(310): Store=6ec5b9b2544c83568def9f0f3c2dac34/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:29,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:29,335 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:29,337 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:29,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:29,340 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:55:29,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6ec5b9b2544c83568def9f0f3c2dac34; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63955637, jitterRate=-0.046986743807792664}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:29,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6ec5b9b2544c83568def9f0f3c2dac34: 2023-04-22 14:55:29,341 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34., pid=127, masterSystemTime=1682175329307 2023-04-22 14:55:29,344 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=6ec5b9b2544c83568def9f0f3c2dac34, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,345 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175329344"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175329344"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175329344"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175329344"}]},"ts":"1682175329344"} 2023-04-22 14:55:29,345 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:29,345 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:29,352 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=126 2023-04-22 14:55:29,352 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure 6ec5b9b2544c83568def9f0f3c2dac34, server=jenkins-hbase4.apache.org,33597,1682175218631 in 191 msec 2023-04-22 14:55:29,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=126, resume processing ppid=125 2023-04-22 14:55:29,355 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=125, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=6ec5b9b2544c83568def9f0f3c2dac34, ASSIGN in 352 msec 2023-04-22 14:55:29,356 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:55:29,357 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175329356"}]},"ts":"1682175329356"} 2023-04-22 14:55:29,363 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=ENABLED in hbase:meta 2023-04-22 14:55:29,368 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=125, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegionEmptyDaughter execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:55:29,369 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=125, state=SUCCESS; CreateTableProcedure table=testSplitTableRegionEmptyDaughter in 419 msec 2023-04-22 14:55:29,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:29,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,465 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-22 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-22 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,465 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,467 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,467 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,467 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,467 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,467 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:29,467 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,470 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:29,470 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:29,492 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,492 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:29,492 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,492 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:29,492 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:29,492 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:29,495 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,496 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:29,504 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:29,504 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:29,508 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:29,508 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. started... 2023-04-22 14:55:29,510 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ec5b9b2544c83568def9f0f3c2dac34 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-22 14:55:29,552 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/7af00b4725bd42a08ccf37619cddebe4 2023-04-22 14:55:29,626 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/fd8645e8d67140e0ad6fcc2ab79c8575 2023-04-22 14:55:29,631 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/7af00b4725bd42a08ccf37619cddebe4 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/7af00b4725bd42a08ccf37619cddebe4 2023-04-22 14:55:29,636 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/7af00b4725bd42a08ccf37619cddebe4, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:55:29,637 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/fd8645e8d67140e0ad6fcc2ab79c8575 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/fd8645e8d67140e0ad6fcc2ab79c8575 2023-04-22 14:55:29,641 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/fd8645e8d67140e0ad6fcc2ab79c8575, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:55:29,642 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 6ec5b9b2544c83568def9f0f3c2dac34 in 132ms, sequenceid=6, compaction requested=false 2023-04-22 14:55:29,642 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:29,643 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ec5b9b2544c83568def9f0f3c2dac34: 2023-04-22 14:55:29,643 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:29,646 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-22 14:55:29,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eec6e41[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:29,651 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-22 14:55:29,651 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,654 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,654 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,654 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:29,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,654 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:29,655 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:29,654 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:29,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,657 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:29,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:55:29,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:55:29,662 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-22 14:55:29,662 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:29,662 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:29,662 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,662 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:29,665 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,665 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,665 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,665 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,665 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,665 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,665 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,666 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:29,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,667 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:29,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,668 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,671 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:29,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:29,672 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,672 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:29,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:29,679 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,679 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,679 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,679 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,679 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,679 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:29,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,680 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,680 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,680 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:29,680 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:29,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,680 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:29,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:29,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:29,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,680 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,681 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:29,681 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,681 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,681 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:29,681 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,681 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:29,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:29,680 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:29,681 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:29,781 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:29,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:29,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:29,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:29,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,800 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:29,800 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,800 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-22 14:55:29,800 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-22 14:55:29,801 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,801 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,802 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,802 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,802 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,803 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:29,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,804 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:29,804 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:29,805 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:29,806 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:29,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:29,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,808 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:29,808 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,808 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:29,808 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,809 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,809 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:29,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:29,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:29,816 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:29,818 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:29,824 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:29,824 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:29,824 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:29,824 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. started... 2023-04-22 14:55:29,824 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ec5b9b2544c83568def9f0f3c2dac34 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:29,875 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:55:29,903 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/eb316a7e422d43d1977f846312e9ce0f 2023-04-22 14:55:29,987 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/db92c2b0cfb545a8bce0701058aaa28a 2023-04-22 14:55:29,992 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/eb316a7e422d43d1977f846312e9ce0f as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/eb316a7e422d43d1977f846312e9ce0f 2023-04-22 14:55:30,007 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/eb316a7e422d43d1977f846312e9ce0f, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:55:30,008 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/db92c2b0cfb545a8bce0701058aaa28a as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/db92c2b0cfb545a8bce0701058aaa28a 2023-04-22 14:55:30,013 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/db92c2b0cfb545a8bce0701058aaa28a, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:55:30,014 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 6ec5b9b2544c83568def9f0f3c2dac34 in 190ms, sequenceid=19, compaction requested=false 2023-04-22 14:55:30,014 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ec5b9b2544c83568def9f0f3c2dac34: 2023-04-22 14:55:30,014 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:30,014 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:30,016 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,016 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,017 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,017 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-22 14:55:30,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1204c7[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:30,020 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-22 14:55:30,020 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:30,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:30,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:30,022 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,022 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:30,022 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:30,023 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:30,023 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:30,026 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:55:30,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:55:30,031 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-22 14:55:30,031 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:30,031 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:30,031 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,031 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:30,032 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,032 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,033 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,032 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,033 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,033 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,033 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,033 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,035 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,035 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,037 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,038 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,038 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:30,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:30,043 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,044 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:30,044 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:30,043 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,044 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,044 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,044 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,044 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,044 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,045 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,045 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,144 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:30,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:30,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:30,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,192 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:30,193 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,193 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-22 14:55:30,193 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-22 14:55:30,193 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,193 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,195 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,195 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,195 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,195 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,197 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,197 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:30,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:30,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,208 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:30,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:30,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,220 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:30,220 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,220 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:30,220 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:30,220 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:30,223 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,223 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,223 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:30,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:30,224 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:30,225 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,226 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,226 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:30,232 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:30,232 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:30,232 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. started... 2023-04-22 14:55:30,242 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ec5b9b2544c83568def9f0f3c2dac34 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:30,303 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/f230d381043045819dc852c96e533e87 2023-04-22 14:55:30,333 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/5da6c9130c8848b0be59b5c01e1a75b7 2023-04-22 14:55:30,345 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/f230d381043045819dc852c96e533e87 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/f230d381043045819dc852c96e533e87 2023-04-22 14:55:30,349 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/f230d381043045819dc852c96e533e87, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:55:30,350 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/5da6c9130c8848b0be59b5c01e1a75b7 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/5da6c9130c8848b0be59b5c01e1a75b7 2023-04-22 14:55:30,362 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/5da6c9130c8848b0be59b5c01e1a75b7, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:55:30,363 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 6ec5b9b2544c83568def9f0f3c2dac34 in 121ms, sequenceid=32, compaction requested=true 2023-04-22 14:55:30,363 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ec5b9b2544c83568def9f0f3c2dac34: 2023-04-22 14:55:30,363 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:30,363 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:30,363 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:30,364 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,364 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:30,364 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:30,365 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-22 14:55:30,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5206fabc[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:30,368 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-22 14:55:30,369 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,371 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,371 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:30,372 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:30,372 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:30,373 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,373 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:30,377 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:30,377 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,377 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:30,379 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:30,379 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,379 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:30,379 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:30,380 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,380 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:55:30,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:55:30,385 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-22 14:55:30,385 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:30,385 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:30,385 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,385 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:30,390 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,390 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,390 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,391 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,391 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,391 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,391 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,392 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,392 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,393 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,396 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,396 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,397 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:30,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,400 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,401 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,401 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,401 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,401 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,400 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,401 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:30,401 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:30,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,501 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:30,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:30,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:30,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:30,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,520 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:30,520 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,520 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-22 14:55:30,520 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-22 14:55:30,521 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,521 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,522 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,522 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,522 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,522 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:30,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,522 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,522 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:30,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,524 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:30,524 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,524 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:30,524 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:30,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:30,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,524 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:30,524 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:30,524 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:30,524 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:30,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:30,525 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:30,525 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:30,525 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. started... 2023-04-22 14:55:30,525 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ec5b9b2544c83568def9f0f3c2dac34 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:30,526 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,528 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,528 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,528 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,528 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,528 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,544 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/b06e1be657054dbda6c4e5c138e88f4f 2023-04-22 14:55:30,964 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/f2d5d7ae6ae5473cb1332f50b2df69ea 2023-04-22 14:55:30,968 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/b06e1be657054dbda6c4e5c138e88f4f as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/b06e1be657054dbda6c4e5c138e88f4f 2023-04-22 14:55:30,972 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/b06e1be657054dbda6c4e5c138e88f4f, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:55:30,973 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/f2d5d7ae6ae5473cb1332f50b2df69ea as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/f2d5d7ae6ae5473cb1332f50b2df69ea 2023-04-22 14:55:30,977 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/f2d5d7ae6ae5473cb1332f50b2df69ea, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:55:30,978 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 6ec5b9b2544c83568def9f0f3c2dac34 in 452ms, sequenceid=45, compaction requested=true 2023-04-22 14:55:30,978 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ec5b9b2544c83568def9f0f3c2dac34: 2023-04-22 14:55:30,978 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:30,978 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:30,978 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:30,978 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,978 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:30,978 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:30,980 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,980 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,981 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,981 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-22 14:55:30,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ceeb859[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:30,983 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-22 14:55:30,983 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,984 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,985 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,985 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:30,985 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:30,985 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,985 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:30,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,985 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:30,985 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:30,985 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:30,986 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:30,986 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:30,986 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:30,987 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,987 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:30,989 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:30,989 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,989 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:30,990 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,990 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:30,990 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,990 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:30,990 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:30,990 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:30,990 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:30,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:30,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:55:30,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:55:30,995 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-22 14:55:30,995 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:30,995 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:30,995 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,995 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:30,998 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,998 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:30,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:30,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:30,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,002 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,002 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,004 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:31,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:31,005 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,006 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:31,006 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,010 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,010 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,010 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,016 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,016 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,016 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,016 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,017 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:31,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,017 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,018 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:31,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,018 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:31,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,118 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:31,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:31,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:31,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:31,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,138 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:31,138 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,138 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-22 14:55:31,138 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-22 14:55:31,138 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,138 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,140 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,140 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,140 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,141 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,141 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,141 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:31,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:31,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:31,143 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:31,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:31,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:31,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:31,145 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:31,145 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. started... 2023-04-22 14:55:31,145 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ec5b9b2544c83568def9f0f3c2dac34 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:31,147 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,148 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,149 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:31,166 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/95fc20ec9b10416a86bfc6f76e3a8df9 2023-04-22 14:55:31,202 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/8fb86187cb8f40e8b5c3df93274d2e15 2023-04-22 14:55:31,209 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/95fc20ec9b10416a86bfc6f76e3a8df9 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/95fc20ec9b10416a86bfc6f76e3a8df9 2023-04-22 14:55:31,215 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/95fc20ec9b10416a86bfc6f76e3a8df9, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:55:31,217 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/8fb86187cb8f40e8b5c3df93274d2e15 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/8fb86187cb8f40e8b5c3df93274d2e15 2023-04-22 14:55:31,222 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/8fb86187cb8f40e8b5c3df93274d2e15, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:55:31,223 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 6ec5b9b2544c83568def9f0f3c2dac34 in 78ms, sequenceid=58, compaction requested=true 2023-04-22 14:55:31,223 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ec5b9b2544c83568def9f0f3c2dac34: 2023-04-22 14:55:31,223 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:31,224 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:31,225 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,225 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:31,226 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,226 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:31,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-22 14:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41ed4be3[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:31,228 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-22 14:55:31,228 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:31,230 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:31,233 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,233 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:31,235 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:31,236 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:31,236 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:31,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:55:31,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:55:31,242 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-22 14:55:31,242 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:31,242 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:31,242 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,242 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:31,244 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,244 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:31,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:31,246 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,246 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:31,246 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:31,248 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,248 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,248 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:31,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:31,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,252 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:31,252 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:31,251 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:31,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,252 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,252 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,251 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:31,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,253 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:31,252 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:31,253 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,253 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:31,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:31,352 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:31,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:31,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:31,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegionEmptyDaughter was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:31,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,367 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:31,367 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,367 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' 2023-04-22 14:55:31,367 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegionEmptyDaughter', kicking off acquire phase on members. 2023-04-22 14:55:31,368 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,368 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,369 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,369 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,369 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,369 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:31,370 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:31,370 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:31,371 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:31,371 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:31,371 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,371 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:31,372 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:31,372 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:31,372 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:31,373 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:31,373 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:31,373 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:31,373 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. started... 2023-04-22 14:55:31,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegionEmptyDaughter is still present, removing 2023-04-22 14:55:31,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,373 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6ec5b9b2544c83568def9f0f3c2dac34 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:31,373 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegionEmptyDaughter' with timeout 60000ms 2023-04-22 14:55:31,373 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:31,373 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegionEmptyDaughter' starting 'acquire' stage 2023-04-22 14:55:31,373 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:31,373 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:31,374 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,374 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,374 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:31,375 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,375 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:31,375 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:31,793 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/8f6bf39cae3746528560f634876b3325 2023-04-22 14:55:32,070 ERROR [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid 0e9964a3-eb0e-4f0a-bd76-2c6599875fd5) service to localhost/127.0.0.1:35843 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:32,206 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/c71548870ea7465a958c28b19eedc137 2023-04-22 14:55:32,211 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/8f6bf39cae3746528560f634876b3325 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/8f6bf39cae3746528560f634876b3325 2023-04-22 14:55:32,215 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/8f6bf39cae3746528560f634876b3325, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:55:32,216 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/c71548870ea7465a958c28b19eedc137 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/c71548870ea7465a958c28b19eedc137 2023-04-22 14:55:32,220 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/c71548870ea7465a958c28b19eedc137, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:55:32,221 INFO [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 6ec5b9b2544c83568def9f0f3c2dac34 in 848ms, sequenceid=71, compaction requested=true 2023-04-22 14:55:32,221 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6ec5b9b2544c83568def9f0f3c2dac34: 2023-04-22 14:55:32,221 DEBUG [rs(jenkins-hbase4.apache.org,33597,1682175218631)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:32,221 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:32,221 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:32,221 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,221 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegionEmptyDaughter' locally acquired 2023-04-22 14:55:32,221 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegionEmptyDaughter) in zk 2023-04-22 14:55:32,223 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,223 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:32,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:32,224 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,224 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegionEmptyDaughter' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:32,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:32,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure 'testSplitTableRegionEmptyDaughter' on coordinator 2023-04-22 14:55:32,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e565b25[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:32,226 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegionEmptyDaughter' starting 'in-barrier' execution. 2023-04-22 14:55:32,226 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,230 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,230 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,231 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:32,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,231 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,231 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegionEmptyDaughter' received 'reached' from coordinator. 2023-04-22 14:55:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegionEmptyDaughter' locally completed 2023-04-22 14:55:32,231 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegionEmptyDaughter' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:32,234 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:32,234 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegionEmptyDaughter' has notified controller of completion 2023-04-22 14:55:32,235 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegionEmptyDaughter' completed. 2023-04-22 14:55:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:32,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:32,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegionEmptyDaughter' member 'jenkins-hbase4.apache.org,33597,1682175218631': 2023-04-22 14:55:32,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' released barrier for procedure'testSplitTableRegionEmptyDaughter', counting down latch. Waiting for 0 more 2023-04-22 14:55:32,239 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegionEmptyDaughter' execution completed 2023-04-22 14:55:32,239 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:32,239 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:32,239 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,239 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionEmptyDaughterincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:32,241 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,241 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:32,241 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,241 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:32,241 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,241 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:32,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:32,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:32,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,241 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:32,242 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:32,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,243 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:32,243 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:32,245 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,245 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:32,246 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:32,248 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:32,248 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:32,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:32,248 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:32,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:32,249 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,249 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,249 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:32,249 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:32,249 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:32,249 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:32,249 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:32,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:32,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:32,249 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,249 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:32,250 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,250 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,250 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:32,250 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,250 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegionEmptyDaughter 2023-04-22 14:55:32,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:32,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:32,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegionEmptyDaughter' 2023-04-22 14:55:32,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:32,250 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:32,250 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegionEmptyDaughter'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:32,250 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:32,350 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:32,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:32,362 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,363 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=6ec5b9b2544c83568def9f0f3c2dac34, daughterA=e5fcd894fc554ac6154c898bcb58d453, daughterB=8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:32,364 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=6ec5b9b2544c83568def9f0f3c2dac34, daughterA=e5fcd894fc554ac6154c898bcb58d453, daughterB=8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:32,365 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=6ec5b9b2544c83568def9f0f3c2dac34, daughterA=e5fcd894fc554ac6154c898bcb58d453, daughterB=8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:32,365 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=128, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=6ec5b9b2544c83568def9f0f3c2dac34, daughterA=e5fcd894fc554ac6154c898bcb58d453, daughterB=8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:32,367 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=6ec5b9b2544c83568def9f0f3c2dac34, UNASSIGN}] 2023-04-22 14:55:32,368 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=129, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=6ec5b9b2544c83568def9f0f3c2dac34, UNASSIGN 2023-04-22 14:55:32,369 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=6ec5b9b2544c83568def9f0f3c2dac34, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:32,370 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175332369"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175332369"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175332369"}]},"ts":"1682175332369"} 2023-04-22 14:55:32,371 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=129, state=RUNNABLE; CloseRegionProcedure 6ec5b9b2544c83568def9f0f3c2dac34, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:55:32,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6ec5b9b2544c83568def9f0f3c2dac34, disabling compactions & flushes 2023-04-22 14:55:32,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:32,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:32,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. after waiting 0 ms 2023-04-22 14:55:32,524 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:32,524 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6ec5b9b2544c83568def9f0f3c2dac34 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-22 14:55:32,534 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/5c99a3c21d01489b976210de6e8407b0 2023-04-22 14:55:32,947 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/be094e065da1432594ea086e1840a2bb 2023-04-22 14:55:32,950 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf1/5c99a3c21d01489b976210de6e8407b0 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/5c99a3c21d01489b976210de6e8407b0 2023-04-22 14:55:32,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/5c99a3c21d01489b976210de6e8407b0, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:55:32,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.tmp/cf2/be094e065da1432594ea086e1840a2bb as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/be094e065da1432594ea086e1840a2bb 2023-04-22 14:55:32,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/be094e065da1432594ea086e1840a2bb, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:55:32,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 6ec5b9b2544c83568def9f0f3c2dac34 in 434ms, sequenceid=82, compaction requested=true 2023-04-22 14:55:32,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-22 14:55:32,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:32,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. 2023-04-22 14:55:32,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6ec5b9b2544c83568def9f0f3c2dac34: 2023-04-22 14:55:32,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,964 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=129 updating hbase:meta row=6ec5b9b2544c83568def9f0f3c2dac34, regionState=CLOSED 2023-04-22 14:55:32,964 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175332964"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175332964"}]},"ts":"1682175332964"} 2023-04-22 14:55:32,966 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=129 2023-04-22 14:55:32,966 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=129, state=SUCCESS; CloseRegionProcedure 6ec5b9b2544c83568def9f0f3c2dac34, server=jenkins-hbase4.apache.org,33597,1682175218631 in 594 msec 2023-04-22 14:55:32,968 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=128 2023-04-22 14:55:32,968 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=6ec5b9b2544c83568def9f0f3c2dac34, UNASSIGN in 599 msec 2023-04-22 14:55:32,976 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=128 splitting 14 storefiles, region=6ec5b9b2544c83568def9f0f3c2dac34, threads=14 2023-04-22 14:55:32,976 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/5da6c9130c8848b0be59b5c01e1a75b7 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,977 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/8fb86187cb8f40e8b5c3df93274d2e15 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,977 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/be094e065da1432594ea086e1840a2bb for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,977 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/c71548870ea7465a958c28b19eedc137 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,977 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/db92c2b0cfb545a8bce0701058aaa28a for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,977 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/f2d5d7ae6ae5473cb1332f50b2df69ea for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,977 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/fd8645e8d67140e0ad6fcc2ab79c8575 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,977 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/5c99a3c21d01489b976210de6e8407b0 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,978 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/7af00b4725bd42a08ccf37619cddebe4 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,978 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/8f6bf39cae3746528560f634876b3325 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,978 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/95fc20ec9b10416a86bfc6f76e3a8df9 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,978 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/eb316a7e422d43d1977f846312e9ce0f for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,978 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/b06e1be657054dbda6c4e5c138e88f4f for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,978 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=128 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/f230d381043045819dc852c96e533e87 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,982 DEBUG [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/c71548870ea7465a958c28b19eedc137, top=false 2023-04-22 14:55:32,983 DEBUG [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/f2d5d7ae6ae5473cb1332f50b2df69ea, top=false 2023-04-22 14:55:32,983 DEBUG [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/be094e065da1432594ea086e1840a2bb, top=false 2023-04-22 14:55:32,983 DEBUG [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/f230d381043045819dc852c96e533e87, top=false 2023-04-22 14:55:32,984 DEBUG [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/95fc20ec9b10416a86bfc6f76e3a8df9, top=false 2023-04-22 14:55:32,992 INFO [StoreFileSplitter-pool-3] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-c71548870ea7465a958c28b19eedc137 for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,992 INFO [StoreFileSplitter-pool-10] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-95fc20ec9b10416a86bfc6f76e3a8df9 for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,995 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/c71548870ea7465a958c28b19eedc137 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:32,997 DEBUG [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/fd8645e8d67140e0ad6fcc2ab79c8575, top=false 2023-04-22 14:55:32,998 DEBUG [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/5da6c9130c8848b0be59b5c01e1a75b7, top=false 2023-04-22 14:55:32,998 DEBUG [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/8fb86187cb8f40e8b5c3df93274d2e15, top=false 2023-04-22 14:55:32,999 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/95fc20ec9b10416a86bfc6f76e3a8df9 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,000 DEBUG [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/b06e1be657054dbda6c4e5c138e88f4f, top=false 2023-04-22 14:55:33,000 DEBUG [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/db92c2b0cfb545a8bce0701058aaa28a, top=false 2023-04-22 14:55:33,001 DEBUG [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/8f6bf39cae3746528560f634876b3325, top=false 2023-04-22 14:55:33,005 DEBUG [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/7af00b4725bd42a08ccf37619cddebe4, top=false 2023-04-22 14:55:33,006 DEBUG [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/5c99a3c21d01489b976210de6e8407b0, top=false 2023-04-22 14:55:33,010 DEBUG [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(713): Will create HFileLink file for hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/eb316a7e422d43d1977f846312e9ce0f, top=false 2023-04-22 14:55:33,016 INFO [StoreFileSplitter-pool-5] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f2d5d7ae6ae5473cb1332f50b2df69ea for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,016 INFO [StoreFileSplitter-pool-2] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-be094e065da1432594ea086e1840a2bb for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,020 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/be094e065da1432594ea086e1840a2bb for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,020 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/f2d5d7ae6ae5473cb1332f50b2df69ea for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,026 INFO [StoreFileSplitter-pool-8] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-7af00b4725bd42a08ccf37619cddebe4 for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,026 INFO [StoreFileSplitter-pool-4] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-db92c2b0cfb545a8bce0701058aaa28a for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,029 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/7af00b4725bd42a08ccf37619cddebe4 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,029 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/db92c2b0cfb545a8bce0701058aaa28a for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,032 INFO [StoreFileSplitter-pool-0] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5da6c9130c8848b0be59b5c01e1a75b7 for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,032 INFO [StoreFileSplitter-pool-9] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8f6bf39cae3746528560f634876b3325 for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,032 INFO [StoreFileSplitter-pool-13] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f230d381043045819dc852c96e533e87 for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,032 INFO [StoreFileSplitter-pool-12] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-eb316a7e422d43d1977f846312e9ce0f for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,035 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/5da6c9130c8848b0be59b5c01e1a75b7 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,036 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/eb316a7e422d43d1977f846312e9ce0f for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,036 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/f230d381043045819dc852c96e533e87 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,039 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/8f6bf39cae3746528560f634876b3325 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,039 INFO [StoreFileSplitter-pool-11] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-b06e1be657054dbda6c4e5c138e88f4f for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,040 INFO [StoreFileSplitter-pool-7] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5c99a3c21d01489b976210de6e8407b0 for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,040 INFO [StoreFileSplitter-pool-1] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8fb86187cb8f40e8b5c3df93274d2e15 for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,040 INFO [StoreFileSplitter-pool-6] regionserver.HRegionFileSystem(742): Created linkFile:hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/.splits/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-fd8645e8d67140e0ad6fcc2ab79c8575 for child: e5fcd894fc554ac6154c898bcb58d453, parent: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,042 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/b06e1be657054dbda6c4e5c138e88f4f for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,043 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/5c99a3c21d01489b976210de6e8407b0 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,043 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/8fb86187cb8f40e8b5c3df93274d2e15 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,043 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=128 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/fd8645e8d67140e0ad6fcc2ab79c8575 for region: 6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:33,044 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=128 split storefiles for region 6ec5b9b2544c83568def9f0f3c2dac34 Daughter A: 14 storefiles, Daughter B: 0 storefiles. 2023-04-22 14:55:33,472 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:55:33,474 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:55:33,475 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.","families":{"info":[{"qualifier":"regioninfo","vlen":67,"tag":[],"timestamp":"1682175333475"},{"qualifier":"splitA","vlen":69,"tag":[],"timestamp":"1682175333475"},{"qualifier":"splitB","vlen":69,"tag":[],"timestamp":"1682175333475"}]},"ts":"1682175333475"} 2023-04-22 14:55:33,476 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175333475"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175333475"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175333475"}]},"ts":"1682175333475"} 2023-04-22 14:55:33,476 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175333475"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175333475"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175333475"}]},"ts":"1682175333475"} 2023-04-22 14:55:33,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=e5fcd894fc554ac6154c898bcb58d453, ASSIGN}, {pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8d02e828b01fa0f263b181cf6c5f65d6, ASSIGN}] 2023-04-22 14:55:33,479 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8d02e828b01fa0f263b181cf6c5f65d6, ASSIGN 2023-04-22 14:55:33,480 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=132, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8d02e828b01fa0f263b181cf6c5f65d6, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,33597,1682175218631; forceNewPlan=false, retain=false 2023-04-22 14:55:33,480 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=e5fcd894fc554ac6154c898bcb58d453, ASSIGN 2023-04-22 14:55:33,481 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=131, ppid=128, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=e5fcd894fc554ac6154c898bcb58d453, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,33597,1682175218631; forceNewPlan=false, retain=false 2023-04-22 14:55:33,630 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-22 14:55:33,632 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=8d02e828b01fa0f263b181cf6c5f65d6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:33,632 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175333632"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175333632"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175333632"}]},"ts":"1682175333632"} 2023-04-22 14:55:33,633 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=133, ppid=132, state=RUNNABLE; OpenRegionProcedure 8d02e828b01fa0f263b181cf6c5f65d6, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:55:33,635 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=e5fcd894fc554ac6154c898bcb58d453, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:33,635 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175333635"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175333635"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175333635"}]},"ts":"1682175333635"} 2023-04-22 14:55:33,636 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=131, state=RUNNABLE; OpenRegionProcedure e5fcd894fc554ac6154c898bcb58d453, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:55:33,787 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6. 2023-04-22 14:55:33,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8d02e828b01fa0f263b181cf6c5f65d6, NAME => 'testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6.', STARTKEY => '71', ENDKEY => ''} 2023-04-22 14:55:33,788 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:33,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter 8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:33,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:33,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:33,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:33,789 INFO [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:33,790 DEBUG [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/cf1 2023-04-22 14:55:33,790 DEBUG [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/cf1 2023-04-22 14:55:33,791 INFO [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8d02e828b01fa0f263b181cf6c5f65d6 columnFamilyName cf1 2023-04-22 14:55:33,791 INFO [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] regionserver.HStore(310): Store=8d02e828b01fa0f263b181cf6c5f65d6/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:33,791 INFO [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:33,792 DEBUG [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/cf2 2023-04-22 14:55:33,792 DEBUG [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/cf2 2023-04-22 14:55:33,792 INFO [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8d02e828b01fa0f263b181cf6c5f65d6 columnFamilyName cf2 2023-04-22 14:55:33,793 INFO [StoreOpener-8d02e828b01fa0f263b181cf6c5f65d6-1] regionserver.HStore(310): Store=8d02e828b01fa0f263b181cf6c5f65d6/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:33,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:33,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:33,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:33,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:33,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8d02e828b01fa0f263b181cf6c5f65d6; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72583802, jitterRate=0.0815829336643219}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:33,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8d02e828b01fa0f263b181cf6c5f65d6: 2023-04-22 14:55:33,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6., pid=133, masterSystemTime=1682175333784 2023-04-22 14:55:33,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6. 2023-04-22 14:55:33,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6. 2023-04-22 14:55:33,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. 2023-04-22 14:55:33,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e5fcd894fc554ac6154c898bcb58d453, NAME => 'testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.', STARTKEY => '', ENDKEY => '71'} 2023-04-22 14:55:33,799 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=132 updating hbase:meta row=8d02e828b01fa0f263b181cf6c5f65d6, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:33,799 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175333799"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175333799"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175333799"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175333799"}]},"ts":"1682175333799"} 2023-04-22 14:55:33,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:33,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegionEmptyDaughter e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:33,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:33,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:33,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:33,801 INFO [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:33,802 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1 2023-04-22 14:55:33,802 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1 2023-04-22 14:55:33,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=133, resume processing ppid=132 2023-04-22 14:55:33,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=133, ppid=132, state=SUCCESS; OpenRegionProcedure 8d02e828b01fa0f263b181cf6c5f65d6, server=jenkins-hbase4.apache.org,33597,1682175218631 in 167 msec 2023-04-22 14:55:33,802 INFO [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e5fcd894fc554ac6154c898bcb58d453 columnFamilyName cf1 2023-04-22 14:55:33,803 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8d02e828b01fa0f263b181cf6c5f65d6, ASSIGN in 323 msec 2023-04-22 14:55:33,808 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5c99a3c21d01489b976210de6e8407b0 2023-04-22 14:55:33,811 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-7af00b4725bd42a08ccf37619cddebe4 2023-04-22 14:55:33,814 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8f6bf39cae3746528560f634876b3325 2023-04-22 14:55:33,816 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-95fc20ec9b10416a86bfc6f76e3a8df9 2023-04-22 14:55:33,819 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-b06e1be657054dbda6c4e5c138e88f4f 2023-04-22 14:55:33,821 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-eb316a7e422d43d1977f846312e9ce0f 2023-04-22 14:55:33,823 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f230d381043045819dc852c96e533e87 2023-04-22 14:55:33,823 INFO [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(310): Store=e5fcd894fc554ac6154c898bcb58d453/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:33,824 INFO [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:33,824 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2 2023-04-22 14:55:33,825 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2 2023-04-22 14:55:33,825 INFO [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e5fcd894fc554ac6154c898bcb58d453 columnFamilyName cf2 2023-04-22 14:55:33,829 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5da6c9130c8848b0be59b5c01e1a75b7 2023-04-22 14:55:33,831 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8fb86187cb8f40e8b5c3df93274d2e15 2023-04-22 14:55:33,834 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-be094e065da1432594ea086e1840a2bb 2023-04-22 14:55:33,835 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-c71548870ea7465a958c28b19eedc137 2023-04-22 14:55:33,838 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-db92c2b0cfb545a8bce0701058aaa28a 2023-04-22 14:55:33,840 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f2d5d7ae6ae5473cb1332f50b2df69ea 2023-04-22 14:55:33,842 DEBUG [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-fd8645e8d67140e0ad6fcc2ab79c8575 2023-04-22 14:55:33,842 INFO [StoreOpener-e5fcd894fc554ac6154c898bcb58d453-1] regionserver.HStore(310): Store=e5fcd894fc554ac6154c898bcb58d453/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:33,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:33,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:33,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegionEmptyDaughter descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:33,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:33,846 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e5fcd894fc554ac6154c898bcb58d453; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65979472, jitterRate=-0.016829252243041992}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:33,846 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e5fcd894fc554ac6154c898bcb58d453: 2023-04-22 14:55:33,847 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453., pid=134, masterSystemTime=1682175333784 2023-04-22 14:55:33,847 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:33,848 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:33,848 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:33,848 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:33,848 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 14:55:33,848 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 35728 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-22 14:55:33,848 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1912): e5fcd894fc554ac6154c898bcb58d453/cf1 is initiating minor compaction (all files) 2023-04-22 14:55:33,848 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1912): e5fcd894fc554ac6154c898bcb58d453/cf2 is initiating minor compaction (all files) 2023-04-22 14:55:33,848 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e5fcd894fc554ac6154c898bcb58d453/cf1 in testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. 2023-04-22 14:55:33,848 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e5fcd894fc554ac6154c898bcb58d453/cf2 in testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. 2023-04-22 14:55:33,848 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-7af00b4725bd42a08ccf37619cddebe4, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-eb316a7e422d43d1977f846312e9ce0f, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f230d381043045819dc852c96e533e87, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-b06e1be657054dbda6c4e5c138e88f4f, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-95fc20ec9b10416a86bfc6f76e3a8df9, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8f6bf39cae3746528560f634876b3325, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5c99a3c21d01489b976210de6e8407b0] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/.tmp, totalSize=34.9 K 2023-04-22 14:55:33,848 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-fd8645e8d67140e0ad6fcc2ab79c8575, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-db92c2b0cfb545a8bce0701058aaa28a, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5da6c9130c8848b0be59b5c01e1a75b7, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f2d5d7ae6ae5473cb1332f50b2df69ea, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8fb86187cb8f40e8b5c3df93274d2e15, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-c71548870ea7465a958c28b19eedc137, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-be094e065da1432594ea086e1840a2bb] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/.tmp, totalSize=34.9 K 2023-04-22 14:55:33,848 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-7af00b4725bd42a08ccf37619cddebe4, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175329454 2023-04-22 14:55:33,848 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-fd8645e8d67140e0ad6fcc2ab79c8575, keycount=2, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175329454 2023-04-22 14:55:33,849 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-eb316a7e422d43d1977f846312e9ce0f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175329783 2023-04-22 14:55:33,849 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-db92c2b0cfb545a8bce0701058aaa28a, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175329783 2023-04-22 14:55:33,849 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f230d381043045819dc852c96e533e87, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175330148 2023-04-22 14:55:33,849 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. 2023-04-22 14:55:33,849 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. 2023-04-22 14:55:33,849 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5da6c9130c8848b0be59b5c01e1a75b7, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175330148 2023-04-22 14:55:33,850 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-b06e1be657054dbda6c4e5c138e88f4f, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175330503 2023-04-22 14:55:33,850 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=131 updating hbase:meta row=e5fcd894fc554ac6154c898bcb58d453, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:33,850 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f2d5d7ae6ae5473cb1332f50b2df69ea, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175330503 2023-04-22 14:55:33,850 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175333850"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175333850"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175333850"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175333850"}]},"ts":"1682175333850"} 2023-04-22 14:55:33,850 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-95fc20ec9b10416a86bfc6f76e3a8df9, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175331120 2023-04-22 14:55:33,850 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8fb86187cb8f40e8b5c3df93274d2e15, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175331120 2023-04-22 14:55:33,850 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8f6bf39cae3746528560f634876b3325, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175331354 2023-04-22 14:55:33,850 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-c71548870ea7465a958c28b19eedc137, keycount=10, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175331354 2023-04-22 14:55:33,851 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5c99a3c21d01489b976210de6e8407b0, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175332352 2023-04-22 14:55:33,851 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] compactions.Compactor(207): Compacting testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-be094e065da1432594ea086e1840a2bb, keycount=8, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175332352 2023-04-22 14:55:33,853 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=134, resume processing ppid=131 2023-04-22 14:55:33,853 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=131, state=SUCCESS; OpenRegionProcedure e5fcd894fc554ac6154c898bcb58d453, server=jenkins-hbase4.apache.org,33597,1682175218631 in 215 msec 2023-04-22 14:55:33,855 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=128 2023-04-22 14:55:33,855 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=128, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=e5fcd894fc554ac6154c898bcb58d453, ASSIGN in 374 msec 2023-04-22 14:55:33,856 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=128, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegionEmptyDaughter, parent=6ec5b9b2544c83568def9f0f3c2dac34, daughterA=e5fcd894fc554ac6154c898bcb58d453, daughterB=8d02e828b01fa0f263b181cf6c5f65d6 in 1.4930 sec 2023-04-22 14:55:33,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33597] ipc.CallRunner(144): callId: 991 service: ClientService methodName: Scan size: 157 connection: 172.31.14.131:39428 deadline: 1682175393864, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34. is not online on jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:33,867 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] throttle.PressureAwareThroughputController(145): e5fcd894fc554ac6154c898bcb58d453#cf2#compaction#126 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:33,867 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] throttle.PressureAwareThroughputController(145): e5fcd894fc554ac6154c898bcb58d453#cf1#compaction#127 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:33,877 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/.tmp/cf2/fc2b9efe0ded4f25ac6da458e82b986b as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/fc2b9efe0ded4f25ac6da458e82b986b 2023-04-22 14:55:33,877 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/.tmp/cf1/111d7bc1ce3d4d5c9ecd25d07b747c6c as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/111d7bc1ce3d4d5c9ecd25d07b747c6c 2023-04-22 14:55:33,881 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in e5fcd894fc554ac6154c898bcb58d453/cf1 of e5fcd894fc554ac6154c898bcb58d453 into 111d7bc1ce3d4d5c9ecd25d07b747c6c(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:33,881 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in e5fcd894fc554ac6154c898bcb58d453/cf2 of e5fcd894fc554ac6154c898bcb58d453 into fc2b9efe0ded4f25ac6da458e82b986b(size=7.3 K), total size for store is 7.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:33,881 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e5fcd894fc554ac6154c898bcb58d453: 2023-04-22 14:55:33,881 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e5fcd894fc554ac6154c898bcb58d453: 2023-04-22 14:55:33,881 INFO [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453., storeName=e5fcd894fc554ac6154c898bcb58d453/cf1, priority=9, startTime=1682175333847; duration=0sec 2023-04-22 14:55:33,881 INFO [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453., storeName=e5fcd894fc554ac6154c898bcb58d453/cf2, priority=9, startTime=1682175333848; duration=0sec 2023-04-22 14:55:33,882 DEBUG [RS:2;jenkins-hbase4:33597-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:33,882 DEBUG [RS:2;jenkins-hbase4:33597-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:33,974 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '71', inclusive 2023-04-22 14:55:33,977 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:55:33,978 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testSplitTableRegionEmptyDaughter 2023-04-22 14:55:33,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegionEmptyDaughter 2023-04-22 14:55:33,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=135, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-22 14:55:33,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-04-22 14:55:33,982 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175333982"}]},"ts":"1682175333982"} 2023-04-22 14:55:33,983 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLING in hbase:meta 2023-04-22 14:55:33,984 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegionEmptyDaughter to state=DISABLING 2023-04-22 14:55:33,984 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=e5fcd894fc554ac6154c898bcb58d453, UNASSIGN}, {pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8d02e828b01fa0f263b181cf6c5f65d6, UNASSIGN}] 2023-04-22 14:55:33,985 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8d02e828b01fa0f263b181cf6c5f65d6, UNASSIGN 2023-04-22 14:55:33,985 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=8d02e828b01fa0f263b181cf6c5f65d6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:33,986 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175333985"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175333985"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175333985"}]},"ts":"1682175333985"} 2023-04-22 14:55:33,986 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=138, ppid=137, state=RUNNABLE; CloseRegionProcedure 8d02e828b01fa0f263b181cf6c5f65d6, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:55:33,988 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=135, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=e5fcd894fc554ac6154c898bcb58d453, UNASSIGN 2023-04-22 14:55:33,989 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=e5fcd894fc554ac6154c898bcb58d453, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:33,989 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175333989"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175333989"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175333989"}]},"ts":"1682175333989"} 2023-04-22 14:55:33,990 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=139, ppid=136, state=RUNNABLE; CloseRegionProcedure e5fcd894fc554ac6154c898bcb58d453, server=jenkins-hbase4.apache.org,33597,1682175218631}] 2023-04-22 14:55:34,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-04-22 14:55:34,138 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:34,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8d02e828b01fa0f263b181cf6c5f65d6, disabling compactions & flushes 2023-04-22 14:55:34,139 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6. 2023-04-22 14:55:34,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6. 2023-04-22 14:55:34,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6. after waiting 0 ms 2023-04-22 14:55:34,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6. 2023-04-22 14:55:34,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/recovered.edits/88.seqid, newMaxSeqId=88, maxSeqId=85 2023-04-22 14:55:34,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:34,143 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6. 2023-04-22 14:55:34,143 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8d02e828b01fa0f263b181cf6c5f65d6: 2023-04-22 14:55:34,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:34,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:34,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e5fcd894fc554ac6154c898bcb58d453, disabling compactions & flushes 2023-04-22 14:55:34,144 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. 2023-04-22 14:55:34,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. 2023-04-22 14:55:34,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. after waiting 0 ms 2023-04-22 14:55:34,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. 2023-04-22 14:55:34,144 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=8d02e828b01fa0f263b181cf6c5f65d6, regionState=CLOSED 2023-04-22 14:55:34,144 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175334144"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175334144"}]},"ts":"1682175334144"} 2023-04-22 14:55:34,145 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-7af00b4725bd42a08ccf37619cddebe4, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-eb316a7e422d43d1977f846312e9ce0f, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f230d381043045819dc852c96e533e87, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-b06e1be657054dbda6c4e5c138e88f4f, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-95fc20ec9b10416a86bfc6f76e3a8df9, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8f6bf39cae3746528560f634876b3325, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5c99a3c21d01489b976210de6e8407b0] to archive 2023-04-22 14:55:34,145 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:34,147 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=138, resume processing ppid=137 2023-04-22 14:55:34,147 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=137, state=SUCCESS; CloseRegionProcedure 8d02e828b01fa0f263b181cf6c5f65d6, server=jenkins-hbase4.apache.org,33597,1682175218631 in 159 msec 2023-04-22 14:55:34,147 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-7af00b4725bd42a08ccf37619cddebe4 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-7af00b4725bd42a08ccf37619cddebe4 2023-04-22 14:55:34,148 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=8d02e828b01fa0f263b181cf6c5f65d6, UNASSIGN in 163 msec 2023-04-22 14:55:34,149 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-eb316a7e422d43d1977f846312e9ce0f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-eb316a7e422d43d1977f846312e9ce0f 2023-04-22 14:55:34,150 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f230d381043045819dc852c96e533e87 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f230d381043045819dc852c96e533e87 2023-04-22 14:55:34,151 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-b06e1be657054dbda6c4e5c138e88f4f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-b06e1be657054dbda6c4e5c138e88f4f 2023-04-22 14:55:34,152 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-95fc20ec9b10416a86bfc6f76e3a8df9 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-95fc20ec9b10416a86bfc6f76e3a8df9 2023-04-22 14:55:34,153 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8f6bf39cae3746528560f634876b3325 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8f6bf39cae3746528560f634876b3325 2023-04-22 14:55:34,154 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5c99a3c21d01489b976210de6e8407b0 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5c99a3c21d01489b976210de6e8407b0 2023-04-22 14:55:34,155 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-fd8645e8d67140e0ad6fcc2ab79c8575, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-db92c2b0cfb545a8bce0701058aaa28a, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5da6c9130c8848b0be59b5c01e1a75b7, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f2d5d7ae6ae5473cb1332f50b2df69ea, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8fb86187cb8f40e8b5c3df93274d2e15, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-c71548870ea7465a958c28b19eedc137, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-be094e065da1432594ea086e1840a2bb] to archive 2023-04-22 14:55:34,155 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:34,157 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-fd8645e8d67140e0ad6fcc2ab79c8575 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-fd8645e8d67140e0ad6fcc2ab79c8575 2023-04-22 14:55:34,157 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-db92c2b0cfb545a8bce0701058aaa28a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-db92c2b0cfb545a8bce0701058aaa28a 2023-04-22 14:55:34,158 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5da6c9130c8848b0be59b5c01e1a75b7 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-5da6c9130c8848b0be59b5c01e1a75b7 2023-04-22 14:55:34,159 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f2d5d7ae6ae5473cb1332f50b2df69ea to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-f2d5d7ae6ae5473cb1332f50b2df69ea 2023-04-22 14:55:34,159 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8fb86187cb8f40e8b5c3df93274d2e15 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-8fb86187cb8f40e8b5c3df93274d2e15 2023-04-22 14:55:34,160 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-c71548870ea7465a958c28b19eedc137 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-c71548870ea7465a958c28b19eedc137 2023-04-22 14:55:34,161 DEBUG [StoreCloser-testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-be094e065da1432594ea086e1840a2bb to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/testSplitTableRegionEmptyDaughter=6ec5b9b2544c83568def9f0f3c2dac34-be094e065da1432594ea086e1840a2bb 2023-04-22 14:55:34,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:55:34,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:34,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453. 2023-04-22 14:55:34,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e5fcd894fc554ac6154c898bcb58d453: 2023-04-22 14:55:34,166 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:34,166 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=e5fcd894fc554ac6154c898bcb58d453, regionState=CLOSED 2023-04-22 14:55:34,166 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1682175334166"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175334166"}]},"ts":"1682175334166"} 2023-04-22 14:55:34,168 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=139, resume processing ppid=136 2023-04-22 14:55:34,168 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=136, state=SUCCESS; CloseRegionProcedure e5fcd894fc554ac6154c898bcb58d453, server=jenkins-hbase4.apache.org,33597,1682175218631 in 177 msec 2023-04-22 14:55:34,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=136, resume processing ppid=135 2023-04-22 14:55:34,169 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=135, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegionEmptyDaughter, region=e5fcd894fc554ac6154c898bcb58d453, UNASSIGN in 184 msec 2023-04-22 14:55:34,170 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175334170"}]},"ts":"1682175334170"} 2023-04-22 14:55:34,171 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegionEmptyDaughter, state=DISABLED in hbase:meta 2023-04-22 14:55:34,174 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegionEmptyDaughter to state=DISABLED 2023-04-22 14:55:34,175 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=135, state=SUCCESS; DisableTableProcedure table=testSplitTableRegionEmptyDaughter in 195 msec 2023-04-22 14:55:34,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=135 2023-04-22 14:55:34,284 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 135 completed 2023-04-22 14:55:34,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegionEmptyDaughter 2023-04-22 14:55:34,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-22 14:55:34,287 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=140, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-22 14:55:34,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=140 2023-04-22 14:55:34,287 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=140, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-22 14:55:34,291 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:34,291 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:34,291 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:34,293 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/recovered.edits] 2023-04-22 14:55:34,293 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/recovered.edits] 2023-04-22 14:55:34,293 DEBUG [HFileArchiver-10] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/recovered.edits] 2023-04-22 14:55:34,296 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/5c99a3c21d01489b976210de6e8407b0 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/5c99a3c21d01489b976210de6e8407b0 2023-04-22 14:55:34,296 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/111d7bc1ce3d4d5c9ecd25d07b747c6c to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf1/111d7bc1ce3d4d5c9ecd25d07b747c6c 2023-04-22 14:55:34,297 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/7af00b4725bd42a08ccf37619cddebe4 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/7af00b4725bd42a08ccf37619cddebe4 2023-04-22 14:55:34,298 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/fc2b9efe0ded4f25ac6da458e82b986b to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/cf2/fc2b9efe0ded4f25ac6da458e82b986b 2023-04-22 14:55:34,298 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/recovered.edits/88.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6/recovered.edits/88.seqid 2023-04-22 14:55:34,298 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/8f6bf39cae3746528560f634876b3325 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/8f6bf39cae3746528560f634876b3325 2023-04-22 14:55:34,299 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/8d02e828b01fa0f263b181cf6c5f65d6 2023-04-22 14:55:34,300 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/95fc20ec9b10416a86bfc6f76e3a8df9 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/95fc20ec9b10416a86bfc6f76e3a8df9 2023-04-22 14:55:34,301 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/b06e1be657054dbda6c4e5c138e88f4f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/b06e1be657054dbda6c4e5c138e88f4f 2023-04-22 14:55:34,301 DEBUG [HFileArchiver-10] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453/recovered.edits/90.seqid 2023-04-22 14:55:34,301 DEBUG [HFileArchiver-10] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/e5fcd894fc554ac6154c898bcb58d453 2023-04-22 14:55:34,302 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/eb316a7e422d43d1977f846312e9ce0f to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/eb316a7e422d43d1977f846312e9ce0f 2023-04-22 14:55:34,303 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/f230d381043045819dc852c96e533e87 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf1/f230d381043045819dc852c96e533e87 2023-04-22 14:55:34,305 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/5da6c9130c8848b0be59b5c01e1a75b7 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/5da6c9130c8848b0be59b5c01e1a75b7 2023-04-22 14:55:34,305 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/8fb86187cb8f40e8b5c3df93274d2e15 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/8fb86187cb8f40e8b5c3df93274d2e15 2023-04-22 14:55:34,306 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/be094e065da1432594ea086e1840a2bb to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/be094e065da1432594ea086e1840a2bb 2023-04-22 14:55:34,307 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/c71548870ea7465a958c28b19eedc137 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/c71548870ea7465a958c28b19eedc137 2023-04-22 14:55:34,308 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/db92c2b0cfb545a8bce0701058aaa28a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/db92c2b0cfb545a8bce0701058aaa28a 2023-04-22 14:55:34,309 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/f2d5d7ae6ae5473cb1332f50b2df69ea to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/f2d5d7ae6ae5473cb1332f50b2df69ea 2023-04-22 14:55:34,310 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/fd8645e8d67140e0ad6fcc2ab79c8575 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/cf2/fd8645e8d67140e0ad6fcc2ab79c8575 2023-04-22 14:55:34,311 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/recovered.edits/85.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34/recovered.edits/85.seqid 2023-04-22 14:55:34,312 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegionEmptyDaughter/6ec5b9b2544c83568def9f0f3c2dac34 2023-04-22 14:55:34,312 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegionEmptyDaughter regions 2023-04-22 14:55:34,314 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=140, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-22 14:55:34,315 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegionEmptyDaughter from hbase:meta 2023-04-22 14:55:34,318 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegionEmptyDaughter' descriptor. 2023-04-22 14:55:34,318 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=140, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-22 14:55:34,319 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegionEmptyDaughter' from region states. 2023-04-22 14:55:34,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175334319"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:34,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175334319"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:34,319 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175334319"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:34,320 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-22 14:55:34,320 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6ec5b9b2544c83568def9f0f3c2dac34, NAME => 'testSplitTableRegionEmptyDaughter,,1682175328949.6ec5b9b2544c83568def9f0f3c2dac34.', STARTKEY => '', ENDKEY => ''}, {ENCODED => e5fcd894fc554ac6154c898bcb58d453, NAME => 'testSplitTableRegionEmptyDaughter,,1682175332362.e5fcd894fc554ac6154c898bcb58d453.', STARTKEY => '', ENDKEY => '71'}, {ENCODED => 8d02e828b01fa0f263b181cf6c5f65d6, NAME => 'testSplitTableRegionEmptyDaughter,71,1682175332362.8d02e828b01fa0f263b181cf6c5f65d6.', STARTKEY => '71', ENDKEY => ''}] 2023-04-22 14:55:34,320 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegionEmptyDaughter' as deleted. 2023-04-22 14:55:34,321 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegionEmptyDaughter","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175334320"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:34,322 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegionEmptyDaughter state from META 2023-04-22 14:55:34,323 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=140, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter 2023-04-22 14:55:34,324 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=140, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegionEmptyDaughter in 38 msec 2023-04-22 14:55:34,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=140 2023-04-22 14:55:34,389 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegionEmptyDaughter, procId: 140 completed 2023-04-22 14:55:34,403 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegionEmptyDaughter Thread=372 (was 374), OpenFileDescriptor=835 (was 833) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=342 (was 337) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=12369 (was 12365) - AvailableMemoryMB LEAK? - 2023-04-22 14:55:34,416 INFO [Listener at localhost/43403] hbase.ResourceChecker(147): before: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=372, OpenFileDescriptor=835, MaxFileDescriptor=60000, SystemLoadAverage=342, ProcessCount=170, AvailableMemoryMB=12368 2023-04-22 14:55:34,416 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:55:34,418 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/balancer 2023-04-22 14:55:34,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(492): Client=jenkins//172.31.14.131 set balanceSwitch=false 2023-04-22 14:55:34,419 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testSplitTableRegion 2023-04-22 14:55:34,420 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_PRE_OPERATION 2023-04-22 14:55:34,421 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-22 14:55:34,423 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,424 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027 empty. 2023-04-22 14:55:34,425 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,425 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-04-22 14:55:34,435 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/.tabledesc/.tableinfo.0000000001 2023-04-22 14:55:34,436 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(7675): creating {ENCODED => 953631f39eb97e99d4a593add584f027, NAME => 'testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='testSplitTableRegion', {NAME => 'cf1', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'cf2', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp 2023-04-22 14:55:34,443 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:34,443 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1604): Closing 953631f39eb97e99d4a593add584f027, disabling compactions & flushes 2023-04-22 14:55:34,443 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:34,443 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:34,443 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. after waiting 0 ms 2023-04-22 14:55:34,443 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:34,443 INFO [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1838): Closed testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:34,443 DEBUG [RegionOpenAndInit-testSplitTableRegion-pool-0] regionserver.HRegion(1558): Region close journal for 953631f39eb97e99d4a593add584f027: 2023-04-22 14:55:34,445 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ADD_TO_META 2023-04-22 14:55:34,446 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175334446"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175334446"}]},"ts":"1682175334446"} 2023-04-22 14:55:34,447 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-22 14:55:34,448 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-22 14:55:34,448 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175334448"}]},"ts":"1682175334448"} 2023-04-22 14:55:34,449 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLING in hbase:meta 2023-04-22 14:55:34,453 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-04-22 14:55:34,454 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-04-22 14:55:34,454 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-04-22 14:55:34,454 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-04-22 14:55:34,454 DEBUG [PEWorker-1] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-04-22 14:55:34,454 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=953631f39eb97e99d4a593add584f027, ASSIGN}] 2023-04-22 14:55:34,456 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=953631f39eb97e99d4a593add584f027, ASSIGN 2023-04-22 14:55:34,457 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=142, ppid=141, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=953631f39eb97e99d4a593add584f027, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:55:34,607 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-04-22 14:55:34,608 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=953631f39eb97e99d4a593add584f027, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,608 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175334608"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175334608"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175334608"}]},"ts":"1682175334608"} 2023-04-22 14:55:34,610 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=142, state=RUNNABLE; OpenRegionProcedure 953631f39eb97e99d4a593add584f027, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:34,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:34,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 953631f39eb97e99d4a593add584f027, NAME => 'testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.', STARTKEY => '', ENDKEY => ''} 2023-04-22 14:55:34,765 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:34,765 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:34,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,766 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,767 INFO [StoreOpener-953631f39eb97e99d4a593add584f027-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,769 DEBUG [StoreOpener-953631f39eb97e99d4a593add584f027-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1 2023-04-22 14:55:34,769 DEBUG [StoreOpener-953631f39eb97e99d4a593add584f027-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1 2023-04-22 14:55:34,769 INFO [StoreOpener-953631f39eb97e99d4a593add584f027-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 953631f39eb97e99d4a593add584f027 columnFamilyName cf1 2023-04-22 14:55:34,770 INFO [StoreOpener-953631f39eb97e99d4a593add584f027-1] regionserver.HStore(310): Store=953631f39eb97e99d4a593add584f027/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:34,770 INFO [StoreOpener-953631f39eb97e99d4a593add584f027-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,771 DEBUG [StoreOpener-953631f39eb97e99d4a593add584f027-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2 2023-04-22 14:55:34,771 DEBUG [StoreOpener-953631f39eb97e99d4a593add584f027-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2 2023-04-22 14:55:34,771 INFO [StoreOpener-953631f39eb97e99d4a593add584f027-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 953631f39eb97e99d4a593add584f027 columnFamilyName cf2 2023-04-22 14:55:34,772 INFO [StoreOpener-953631f39eb97e99d4a593add584f027-1] regionserver.HStore(310): Store=953631f39eb97e99d4a593add584f027/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:34,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:34,775 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:34,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-22 14:55:34,777 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 953631f39eb97e99d4a593add584f027; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73229063, jitterRate=0.09119807183742523}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:34,777 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 953631f39eb97e99d4a593add584f027: 2023-04-22 14:55:34,778 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027., pid=143, masterSystemTime=1682175334761 2023-04-22 14:55:34,779 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:34,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:34,779 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=142 updating hbase:meta row=953631f39eb97e99d4a593add584f027, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,779 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175334779"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175334779"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175334779"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175334779"}]},"ts":"1682175334779"} 2023-04-22 14:55:34,781 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=142 2023-04-22 14:55:34,782 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=142, state=SUCCESS; OpenRegionProcedure 953631f39eb97e99d4a593add584f027, server=jenkins-hbase4.apache.org,36267,1682175218583 in 170 msec 2023-04-22 14:55:34,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=141 2023-04-22 14:55:34,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=141, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=953631f39eb97e99d4a593add584f027, ASSIGN in 328 msec 2023-04-22 14:55:34,783 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-22 14:55:34,783 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175334783"}]},"ts":"1682175334783"} 2023-04-22 14:55:34,784 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=ENABLED in hbase:meta 2023-04-22 14:55:34,786 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=141, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testSplitTableRegion execute state=CREATE_TABLE_POST_OPERATION 2023-04-22 14:55:34,787 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=141, state=SUCCESS; CreateTableProcedure table=testSplitTableRegion in 368 msec 2023-04-22 14:55:34,818 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-22 14:55:34,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:34,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-22 14:55:34,927 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-22 14:55:34,927 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:34,927 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-22 14:55:34,927 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-22 14:55:34,928 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,928 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:34,929 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:34,929 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:34,929 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:34,929 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:34,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:34,929 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,929 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:34,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:34,930 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:34,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:34,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:34,931 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:34,931 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:34,931 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. started... 2023-04-22 14:55:34,932 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 953631f39eb97e99d4a593add584f027 2/2 column families, dataSize=120 B heapSize=928 B 2023-04-22 14:55:34,932 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,933 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,933 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,933 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:34,934 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,934 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:34,941 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7 2023-04-22 14:55:34,952 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/8909126e91094f838230b62af77056a1 2023-04-22 14:55:34,956 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7 2023-04-22 14:55:34,959 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:55:34,960 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/8909126e91094f838230b62af77056a1 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1 2023-04-22 14:55:34,962 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:55:34,963 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120 B/120, heapSize ~896 B/896, currentSize=0 B/0 for 953631f39eb97e99d4a593add584f027 in 31ms, sequenceid=6, compaction requested=false 2023-04-22 14:55:34,963 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testSplitTableRegion' 2023-04-22 14:55:34,964 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 953631f39eb97e99d4a593add584f027: 2023-04-22 14:55:34,964 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:34,964 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:34,966 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,966 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:34,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-22 14:55:34,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b609b49[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:34,968 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-22 14:55:34,968 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,969 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,969 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,969 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:34,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:34,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,969 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,970 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:34,969 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:34,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:34,970 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:34,971 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:34,972 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:34,976 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:34,976 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:34,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:34,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-22 14:55:34,979 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-22 14:55:34,979 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:34,979 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:34,979 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-22 14:55:34,979 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:34,981 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,981 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,981 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,981 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:34,981 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:34,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:34,981 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:34,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,982 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,982 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:34,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:34,982 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:34,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:34,983 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:34,984 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:34,984 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:34,984 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:34,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:34,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:34,987 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:34,987 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:34,987 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:34,987 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:34,987 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:34,987 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:34,987 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:34,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-22 14:55:34,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:34,987 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:34,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:34,987 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:34,987 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:34,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:34,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:34,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:34,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:34,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:34,988 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,088 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:35,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:35,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:35,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:35,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-22 14:55:35,099 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-22 14:55:35,099 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,099 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-22 14:55:35,099 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-22 14:55:35,100 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,100 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,101 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,101 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,101 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,101 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,101 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,101 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,103 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,103 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,103 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. started... 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,103 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,103 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 953631f39eb97e99d4a593add584f027 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,106 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,107 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,112 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/da9ab957c82e4db78955d94c0ec8ad41 2023-04-22 14:55:35,123 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=19 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/f20c2072c13140ef97d30bd7a5bab886 2023-04-22 14:55:35,126 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/da9ab957c82e4db78955d94c0ec8ad41 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41 2023-04-22 14:55:35,130 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:55:35,130 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/f20c2072c13140ef97d30bd7a5bab886 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886 2023-04-22 14:55:35,134 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886, entries=10, sequenceid=19, filesize=5.0 K 2023-04-22 14:55:35,135 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 953631f39eb97e99d4a593add584f027 in 32ms, sequenceid=19, compaction requested=false 2023-04-22 14:55:35,135 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 953631f39eb97e99d4a593add584f027: 2023-04-22 14:55:35,135 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,137 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-22 14:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@265d20c[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:35,139 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-22 14:55:35,139 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,140 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,141 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,141 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:35,140 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,141 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,140 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,141 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,141 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:35,141 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:35,141 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,142 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,142 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,144 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,144 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,144 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:35,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-22 14:55:35,147 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-22 14:55:35,147 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:35,147 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:35,147 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-22 14:55:35,147 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:35,148 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,148 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,148 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,148 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,148 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,148 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,149 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,149 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,149 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,150 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,151 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,151 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,152 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,154 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,154 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,154 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,154 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,154 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,154 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,154 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,154 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-22 14:55:35,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:35,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,154 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,154 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,155 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,155 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:35,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,155 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,155 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,155 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:35,155 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,155 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,155 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,156 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,156 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,255 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:35,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:35,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:35,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:35,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-22 14:55:35,268 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-22 14:55:35,268 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,268 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-22 14:55:35,268 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-22 14:55:35,268 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,268 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,269 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,269 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,269 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,270 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,270 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,270 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,271 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,272 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,272 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,272 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,272 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,272 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,272 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,272 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,272 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:35,272 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:35,272 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,272 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. started... 2023-04-22 14:55:35,272 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 953631f39eb97e99d4a593add584f027 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,278 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,284 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/3a7d8eb9d97d43cf824a628a587e5033 2023-04-22 14:55:35,294 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=32 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/8a1d7046d8cc46499032d76c974c6a4a 2023-04-22 14:55:35,297 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/3a7d8eb9d97d43cf824a628a587e5033 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033 2023-04-22 14:55:35,300 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:55:35,301 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/8a1d7046d8cc46499032d76c974c6a4a as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a 2023-04-22 14:55:35,304 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a, entries=10, sequenceid=32, filesize=5.0 K 2023-04-22 14:55:35,305 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 953631f39eb97e99d4a593add584f027 in 33ms, sequenceid=32, compaction requested=true 2023-04-22 14:55:35,305 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 953631f39eb97e99d4a593add584f027: 2023-04-22 14:55:35,305 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,305 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:35,305 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:35,305 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,305 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,305 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,306 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,306 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,307 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,307 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-22 14:55:35,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@297c23f0[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:35,308 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-22 14:55:35,308 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,309 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,309 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,309 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,309 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,310 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,310 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,310 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,310 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:35,310 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,310 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:35,310 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,310 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,310 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:35,310 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,311 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,311 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:35,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-22 14:55:35,314 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-22 14:55:35,314 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:35,314 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:35,314 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-22 14:55:35,314 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:35,316 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,316 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,316 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,316 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,316 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,316 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,317 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,317 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,317 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,318 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,319 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,319 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,320 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,322 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,322 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,322 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,322 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,322 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,322 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,322 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,322 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-22 14:55:35,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,322 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,322 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:35,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:35,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,323 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,423 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:35,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:35,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:35,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:35,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-22 14:55:35,435 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-22 14:55:35,435 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,435 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-22 14:55:35,435 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-22 14:55:35,436 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,436 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,437 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,437 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,437 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,437 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,437 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,437 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:35,439 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:35,439 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,439 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. started... 2023-04-22 14:55:35,439 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 953631f39eb97e99d4a593add584f027 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:35,440 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,441 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,441 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,441 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,441 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,442 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,448 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/57e1413d85f749e0bbbf5ee90269d4c4 2023-04-22 14:55:35,458 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/e691c55c7f0f4b7ca99f5f0e4271c226 2023-04-22 14:55:35,461 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/57e1413d85f749e0bbbf5ee90269d4c4 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4 2023-04-22 14:55:35,464 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:55:35,465 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/e691c55c7f0f4b7ca99f5f0e4271c226 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226 2023-04-22 14:55:35,468 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226, entries=10, sequenceid=45, filesize=5.0 K 2023-04-22 14:55:35,468 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 953631f39eb97e99d4a593add584f027 in 29ms, sequenceid=45, compaction requested=true 2023-04-22 14:55:35,468 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 953631f39eb97e99d4a593add584f027: 2023-04-22 14:55:35,468 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:35,468 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,469 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,469 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,471 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,471 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,471 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,471 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-22 14:55:35,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@550fda89[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:35,473 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-22 14:55:35,473 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,474 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:35,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:35,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,474 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:35,475 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,475 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,475 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,476 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,476 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,477 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,477 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,477 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:35,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-22 14:55:35,479 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-22 14:55:35,479 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:35,479 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:35,479 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-22 14:55:35,480 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:35,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,483 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,488 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,488 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,489 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,490 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,491 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,491 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,495 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,496 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-22 14:55:35,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,496 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:35,497 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,497 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,497 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:35,497 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,497 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,497 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,597 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:35,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:35,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:35,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:35,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-22 14:55:35,610 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-22 14:55:35,610 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,610 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-22 14:55:35,610 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-22 14:55:35,610 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,610 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,612 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,612 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,612 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,612 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,612 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,612 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,613 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:35,614 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,614 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. started... 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,614 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,614 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 953631f39eb97e99d4a593add584f027 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:35,616 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,617 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,617 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,617 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,617 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,617 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,622 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/6852b591f3f8423bbde72c5d7711d24e 2023-04-22 14:55:35,633 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/bec8e24e22fc46fdbe734863ca69db57 2023-04-22 14:55:35,637 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/6852b591f3f8423bbde72c5d7711d24e as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e 2023-04-22 14:55:35,640 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:55:35,640 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/bec8e24e22fc46fdbe734863ca69db57 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57 2023-04-22 14:55:35,644 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57, entries=10, sequenceid=58, filesize=5.0 K 2023-04-22 14:55:35,644 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 953631f39eb97e99d4a593add584f027 in 30ms, sequenceid=58, compaction requested=true 2023-04-22 14:55:35,644 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 953631f39eb97e99d4a593add584f027: 2023-04-22 14:55:35,645 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,646 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,646 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,647 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,647 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-22 14:55:35,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1aa9b2[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:35,648 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-22 14:55:35,648 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,649 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,649 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,649 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,649 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,649 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,649 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:35,649 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,649 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,650 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,650 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,650 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:35,650 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:35,652 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,652 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,652 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,652 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:35,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-22 14:55:35,655 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-22 14:55:35,655 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:35,655 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:35,655 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-22 14:55:35,655 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:35,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,657 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,658 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,658 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,659 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,660 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,660 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,662 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,662 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,662 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,662 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,662 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-22 14:55:35,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:35,662 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,662 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,662 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,662 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:35,662 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,663 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,763 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:35,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:35,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-22 14:55:35,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(143): Procedure testSplitTableRegion was in running list but was completed. Accepting new attempt. 2023-04-22 14:55:35,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure.ProcedureCoordinator(165): Submitting procedure testSplitTableRegion 2023-04-22 14:55:35,777 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'testSplitTableRegion' 2023-04-22 14:55:35,777 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,777 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'testSplitTableRegion' starting 'acquire' 2023-04-22 14:55:35,777 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'testSplitTableRegion', kicking off acquire phase on members. 2023-04-22 14:55:35,777 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,777 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,779 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,779 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,779 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,779 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,780 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,780 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-22 14:55:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for testSplitTableRegion 2023-04-22 14:55:35,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc testSplitTableRegion is still present, removing 2023-04-22 14:55:35,781 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:testSplitTableRegion 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35417,1682175218529' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'testSplitTableRegion' with timeout 60000ms 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'testSplitTableRegion' starting 'acquire' stage 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,33597,1682175218631' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-22 14:55:35,782 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-22 14:55:35,782 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,782 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. started... 2023-04-22 14:55:35,782 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 953631f39eb97e99d4a593add584f027 2/2 column families, dataSize=600 B heapSize=2.53 KB 2023-04-22 14:55:35,783 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,783 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,784 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,792 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/1c6f0a191e234fa29eb39538e704f5fb 2023-04-22 14:55:35,803 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=300 B at sequenceid=71 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/b593c893effe4bb982f2d54e8b6a5f80 2023-04-22 14:55:35,807 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/1c6f0a191e234fa29eb39538e704f5fb as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb 2023-04-22 14:55:35,810 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:55:35,810 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/b593c893effe4bb982f2d54e8b6a5f80 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80 2023-04-22 14:55:35,813 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80, entries=10, sequenceid=71, filesize=5.0 K 2023-04-22 14:55:35,814 INFO [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~600 B/600, heapSize ~2.50 KB/2560, currentSize=0 B/0 for 953631f39eb97e99d4a593add584f027 in 32ms, sequenceid=71, compaction requested=true 2023-04-22 14:55:35,814 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 953631f39eb97e99d4a593add584f027: 2023-04-22 14:55:35,814 DEBUG [rs(jenkins-hbase4.apache.org,36267,1682175218583)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-22 14:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-22 14:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'testSplitTableRegion' locally acquired 2023-04-22 14:55:35,814 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure (testSplitTableRegion) in zk 2023-04-22 14:55:35,816 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,816 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,816 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,816 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'testSplitTableRegion' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-22 14:55:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,36267,1682175218583' joining acquired barrier for procedure 'testSplitTableRegion' on coordinator 2023-04-22 14:55:35,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5643e14f[Count = 0] remaining members to acquire global barrier 2023-04-22 14:55:35,818 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'testSplitTableRegion' starting 'in-barrier' execution. 2023-04-22 14:55:35,818 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,819 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-22 14:55:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,36267,1682175218583' in zk 2023-04-22 14:55:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,820 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,35417,1682175218529' in zk 2023-04-22 14:55:35,819 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'testSplitTableRegion' received 'reached' from coordinator. 2023-04-22 14:55:35,820 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'testSplitTableRegion' locally completed 2023-04-22 14:55:35,820 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'testSplitTableRegion' completed for member 'jenkins-hbase4.apache.org,33597,1682175218631' in zk 2023-04-22 14:55:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,35417,1682175218529' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,825 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,825 DEBUG [member: 'jenkins-hbase4.apache.org,36267,1682175218583' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,826 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'testSplitTableRegion' has notified controller of completion 2023-04-22 14:55:35,826 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,826 DEBUG [member: 'jenkins-hbase4.apache.org,33597,1682175218631' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'testSplitTableRegion' completed. 2023-04-22 14:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'testSplitTableRegion' member 'jenkins-hbase4.apache.org,36267,1682175218583': 2023-04-22 14:55:35,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,36267,1682175218583' released barrier for procedure'testSplitTableRegion', counting down latch. Waiting for 0 more 2023-04-22 14:55:35,829 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'testSplitTableRegion' execution completed 2023-04-22 14:55:35,829 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-22 14:55:35,829 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-22 14:55:35,829 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:testSplitTableRegion 2023-04-22 14:55:35,829 INFO [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure testSplitTableRegionincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-22 14:55:35,830 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,830 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,830 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,830 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,830 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,830 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-22 14:55:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-22 14:55:35,831 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,831 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-22 14:55:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,831 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,832 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-22 14:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,833 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,834 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-22 14:55:35,834 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----testSplitTableRegion 2023-04-22 14:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,836 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,836 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,836 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,836 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'testSplitTableRegion' 2023-04-22 14:55:35,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-22 14:55:35,836 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,836 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-22 14:55:35,836 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,837 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-22 14:55:35,836 DEBUG [(jenkins-hbase4.apache.org,43497,1682175217266)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-22 14:55:35,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-22 14:55:35,837 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : testSplitTableRegion'' to complete. (max 20000 ms per retry) 2023-04-22 14:55:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-22 14:55:35,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,837 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,837 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-22 14:55:35,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-22 14:55:35,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-22 14:55:35,837 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,837 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/testSplitTableRegion 2023-04-22 14:55:35,838 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:35,838 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,838 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:35,838 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,838 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/testSplitTableRegion 2023-04-22 14:55:35,838 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/testSplitTableRegion 2023-04-22 14:55:35,937 DEBUG [Listener at localhost/43403] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-22 14:55:35,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-22 14:55:35,945 DEBUG [Listener at localhost/43403] assignment.SplitTableRegionProcedure(219): Splittable=true state=OPEN, location=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,946 DEBUG [Listener at localhost/43403] procedure2.ProcedureExecutor(1029): Stored pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=953631f39eb97e99d4a593add584f027, daughterA=b079ecd948cc7c7d6bd91b4d13027ab2, daughterB=ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:35,947 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=953631f39eb97e99d4a593add584f027, daughterA=b079ecd948cc7c7d6bd91b4d13027ab2, daughterB=ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:35,948 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=953631f39eb97e99d4a593add584f027, daughterA=b079ecd948cc7c7d6bd91b4d13027ab2, daughterB=ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:35,948 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=144, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE; SplitTableRegionProcedure table=testSplitTableRegion, parent=953631f39eb97e99d4a593add584f027, daughterA=b079ecd948cc7c7d6bd91b4d13027ab2, daughterB=ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:35,949 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=953631f39eb97e99d4a593add584f027, UNASSIGN}] 2023-04-22 14:55:35,950 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=145, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=953631f39eb97e99d4a593add584f027, UNASSIGN 2023-04-22 14:55:35,951 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=953631f39eb97e99d4a593add584f027, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:35,951 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175335951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175335951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175335951"}]},"ts":"1682175335951"} 2023-04-22 14:55:35,952 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=146, ppid=145, state=RUNNABLE; CloseRegionProcedure 953631f39eb97e99d4a593add584f027, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:36,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 953631f39eb97e99d4a593add584f027, disabling compactions & flushes 2023-04-22 14:55:36,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:36,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:36,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. after waiting 0 ms 2023-04-22 14:55:36,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:36,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 953631f39eb97e99d4a593add584f027 2/2 column families, dataSize=480 B heapSize=2.13 KB 2023-04-22 14:55:36,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/c4074c83fec446079cb2f327b1b93f46 2023-04-22 14:55:36,123 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=240 B at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5 2023-04-22 14:55:36,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf1/c4074c83fec446079cb2f327b1b93f46 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46 2023-04-22 14:55:36,130 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:55:36,131 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/.tmp/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5 2023-04-22 14:55:36,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5, entries=8, sequenceid=82, filesize=5.0 K 2023-04-22 14:55:36,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~480 B/480, heapSize ~2.09 KB/2144, currentSize=0 B/0 for 953631f39eb97e99d4a593add584f027 in 31ms, sequenceid=82, compaction requested=true 2023-04-22 14:55:36,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=1 2023-04-22 14:55:36,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:36,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. 2023-04-22 14:55:36,141 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 953631f39eb97e99d4a593add584f027: 2023-04-22 14:55:36,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,143 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=145 updating hbase:meta row=953631f39eb97e99d4a593add584f027, regionState=CLOSED 2023-04-22 14:55:36,143 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175336143"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175336143"}]},"ts":"1682175336143"} 2023-04-22 14:55:36,145 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=146, resume processing ppid=145 2023-04-22 14:55:36,145 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=146, ppid=145, state=SUCCESS; CloseRegionProcedure 953631f39eb97e99d4a593add584f027, server=jenkins-hbase4.apache.org,36267,1682175218583 in 192 msec 2023-04-22 14:55:36,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=144 2023-04-22 14:55:36,146 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=953631f39eb97e99d4a593add584f027, UNASSIGN in 196 msec 2023-04-22 14:55:36,155 INFO [PEWorker-1] assignment.SplitTableRegionProcedure(694): pid=144 splitting 14 storefiles, region=953631f39eb97e99d4a593add584f027, threads=14 2023-04-22 14:55:36,155 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,155 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,155 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,156 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,156 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,156 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,156 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,156 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,156 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,156 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,156 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,158 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,158 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,158 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(776): pid=144 splitting started for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,311 DEBUG [StoreFileSplitter-pool-6] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,314 DEBUG [StoreFileSplitter-pool-3] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,328 DEBUG [StoreFileSplitter-pool-2] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,339 DEBUG [StoreFileSplitter-pool-4] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,339 DEBUG [StoreFileSplitter-pool-1] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,344 DEBUG [StoreFileSplitter-pool-0] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,356 DEBUG [StoreFileSplitter-pool-9] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,357 DEBUG [StoreFileSplitter-pool-5] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,364 DEBUG [StoreFileSplitter-pool-13] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,371 DEBUG [StoreFileSplitter-pool-10] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,372 DEBUG [StoreFileSplitter-pool-7] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,377 DEBUG [StoreFileSplitter-pool-12] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46 for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,377 DEBUG [StoreFileSplitter-pool-8] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,378 DEBUG [StoreFileSplitter-pool-11] assignment.SplitTableRegionProcedure(787): pid=144 splitting complete for store file: hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e for region: 953631f39eb97e99d4a593add584f027 2023-04-22 14:55:36,378 DEBUG [PEWorker-1] assignment.SplitTableRegionProcedure(755): pid=144 split storefiles for region 953631f39eb97e99d4a593add584f027 Daughter A: 14 storefiles, Daughter B: 14 storefiles. 2023-04-22 14:55:36,413 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:55:36,414 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/recovered.edits/85.seqid, newMaxSeqId=85, maxSeqId=-1 2023-04-22 14:55:36,416 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.","families":{"info":[{"qualifier":"regioninfo","vlen":54,"tag":[],"timestamp":"1682175336415"},{"qualifier":"splitA","vlen":56,"tag":[],"timestamp":"1682175336415"},{"qualifier":"splitB","vlen":56,"tag":[],"timestamp":"1682175336415"}]},"ts":"1682175336415"} 2023-04-22 14:55:36,416 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175336415"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175336415"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175336415"}]},"ts":"1682175336415"} 2023-04-22 14:55:36,416 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175336415"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175336415"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175336415"}]},"ts":"1682175336415"} 2023-04-22 14:55:36,418 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=b079ecd948cc7c7d6bd91b4d13027ab2, ASSIGN}, {pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=ca893cb83296e07cd3d89c3d9a1ef1c1, ASSIGN}] 2023-04-22 14:55:36,419 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=ca893cb83296e07cd3d89c3d9a1ef1c1, ASSIGN 2023-04-22 14:55:36,419 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=148, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=ca893cb83296e07cd3d89c3d9a1ef1c1, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:55:36,420 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testSplitTableRegion, region=b079ecd948cc7c7d6bd91b4d13027ab2, ASSIGN 2023-04-22 14:55:36,420 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=147, ppid=144, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testSplitTableRegion, region=b079ecd948cc7c7d6bd91b4d13027ab2, ASSIGN; state=SPLITTING_NEW, location=jenkins-hbase4.apache.org,36267,1682175218583; forceNewPlan=false, retain=false 2023-04-22 14:55:36,570 INFO [jenkins-hbase4:43497] balancer.BaseLoadBalancer(1545): Reassigned 2 regions. 2 retained the pre-restart assignment. 2023-04-22 14:55:36,571 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=b079ecd948cc7c7d6bd91b4d13027ab2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:36,571 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175336571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175336571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175336571"}]},"ts":"1682175336571"} 2023-04-22 14:55:36,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=147, state=RUNNABLE; OpenRegionProcedure b079ecd948cc7c7d6bd91b4d13027ab2, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:36,575 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=ca893cb83296e07cd3d89c3d9a1ef1c1, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:36,575 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175336575"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175336575"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175336575"}]},"ts":"1682175336575"} 2023-04-22 14:55:36,576 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=150, ppid=148, state=RUNNABLE; OpenRegionProcedure ca893cb83296e07cd3d89c3d9a1ef1c1, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:36,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:36,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ca893cb83296e07cd3d89c3d9a1ef1c1, NAME => 'testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.', STARTKEY => '41', ENDKEY => ''} 2023-04-22 14:55:36,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:36,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:36,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:36,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:36,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:36,729 INFO [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:36,729 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1 2023-04-22 14:55:36,729 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1 2023-04-22 14:55:36,730 INFO [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca893cb83296e07cd3d89c3d9a1ef1c1 columnFamilyName cf1 2023-04-22 14:55:36,740 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7-top 2023-04-22 14:55:36,744 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb-top 2023-04-22 14:55:36,746 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033-top 2023-04-22 14:55:36,749 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4-top 2023-04-22 14:55:36,752 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e-top 2023-04-22 14:55:36,755 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46-top 2023-04-22 14:55:36,758 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41-top 2023-04-22 14:55:36,758 INFO [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(310): Store=ca893cb83296e07cd3d89c3d9a1ef1c1/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:36,758 INFO [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:36,759 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2 2023-04-22 14:55:36,759 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2 2023-04-22 14:55:36,759 INFO [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ca893cb83296e07cd3d89c3d9a1ef1c1 columnFamilyName cf2 2023-04-22 14:55:36,769 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1-top 2023-04-22 14:55:36,772 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a-top 2023-04-22 14:55:36,775 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80-top 2023-04-22 14:55:36,778 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57-top 2023-04-22 14:55:36,781 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5-top 2023-04-22 14:55:36,784 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226-top 2023-04-22 14:55:36,787 DEBUG [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886-top 2023-04-22 14:55:36,787 INFO [StoreOpener-ca893cb83296e07cd3d89c3d9a1ef1c1-1] regionserver.HStore(310): Store=ca893cb83296e07cd3d89c3d9a1ef1c1/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:36,788 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:36,789 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:36,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:36,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:36,791 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ca893cb83296e07cd3d89c3d9a1ef1c1; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59182890, jitterRate=-0.11810621619224548}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:36,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ca893cb83296e07cd3d89c3d9a1ef1c1: 2023-04-22 14:55:36,792 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1., pid=150, masterSystemTime=1682175336724 2023-04-22 14:55:36,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:36,793 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:36,793 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:36,793 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:36,793 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:36,793 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1912): ca893cb83296e07cd3d89c3d9a1ef1c1/cf1 is initiating minor compaction (all files) 2023-04-22 14:55:36,793 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:36,793 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ca893cb83296e07cd3d89c3d9a1ef1c1/cf1 in testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:36,793 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1912): ca893cb83296e07cd3d89c3d9a1ef1c1/cf2 is initiating minor compaction (all files) 2023-04-22 14:55:36,793 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ca893cb83296e07cd3d89c3d9a1ef1c1/cf2 in testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:36,794 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46-top] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/.tmp, totalSize=34.9 K 2023-04-22 14:55:36,794 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5-top] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/.tmp, totalSize=34.9 K 2023-04-22 14:55:36,794 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175334921 2023-04-22 14:55:36,794 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=7, earliestPutTs=1682175334921 2023-04-22 14:55:36,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:36,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:36,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:36,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b079ecd948cc7c7d6bd91b4d13027ab2, NAME => 'testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.', STARTKEY => '', ENDKEY => '41'} 2023-04-22 14:55:36,795 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175335090 2023-04-22 14:55:36,795 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=20, earliestPutTs=1682175335090 2023-04-22 14:55:36,795 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=148 updating hbase:meta row=ca893cb83296e07cd3d89c3d9a1ef1c1, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:36,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(174): System coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro loaded, priority=536870911. 2023-04-22 14:55:36,795 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175336795"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175336795"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175336795"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175336795"}]},"ts":"1682175336795"} 2023-04-22 14:55:36,795 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175335257 2023-04-22 14:55:36,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testSplitTableRegion b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:36,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-22 14:55:36,795 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1682175335257 2023-04-22 14:55:36,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:36,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:36,795 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682175335425 2023-04-22 14:55:36,796 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1682175335425 2023-04-22 14:55:36,796 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682175335599 2023-04-22 14:55:36,796 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1682175335599 2023-04-22 14:55:36,796 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1682175335765 2023-04-22 14:55:36,796 INFO [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf1 of region b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:36,797 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1682175335765 2023-04-22 14:55:36,797 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175335939 2023-04-22 14:55:36,797 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1682175335939 2023-04-22 14:55:36,797 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1 2023-04-22 14:55:36,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=150, resume processing ppid=148 2023-04-22 14:55:36,798 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1 2023-04-22 14:55:36,798 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; OpenRegionProcedure ca893cb83296e07cd3d89c3d9a1ef1c1, server=jenkins-hbase4.apache.org,36267,1682175218583 in 220 msec 2023-04-22 14:55:36,798 INFO [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b079ecd948cc7c7d6bd91b4d13027ab2 columnFamilyName cf1 2023-04-22 14:55:36,799 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=148, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=ca893cb83296e07cd3d89c3d9a1ef1c1, ASSIGN in 380 msec 2023-04-22 14:55:36,808 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] throttle.PressureAwareThroughputController(145): ca893cb83296e07cd3d89c3d9a1ef1c1#cf1#compaction#142 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:36,808 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] throttle.PressureAwareThroughputController(145): ca893cb83296e07cd3d89c3d9a1ef1c1#cf2#compaction#143 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:36,818 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7-bottom 2023-04-22 14:55:36,821 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb-bottom 2023-04-22 14:55:36,823 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/.tmp/cf2/b7a6dcf6e65940f3b380845f7beba9df as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/b7a6dcf6e65940f3b380845f7beba9df 2023-04-22 14:55:36,824 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033-bottom 2023-04-22 14:55:36,829 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4-bottom 2023-04-22 14:55:36,831 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/.tmp/cf1/1e79ad0f6ccb43a99b475895b8552fc3 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/1e79ad0f6ccb43a99b475895b8552fc3 2023-04-22 14:55:36,832 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e-bottom 2023-04-22 14:55:36,835 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ca893cb83296e07cd3d89c3d9a1ef1c1/cf2 of ca893cb83296e07cd3d89c3d9a1ef1c1 into b7a6dcf6e65940f3b380845f7beba9df(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:36,835 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca893cb83296e07cd3d89c3d9a1ef1c1: 2023-04-22 14:55:36,835 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1., storeName=ca893cb83296e07cd3d89c3d9a1ef1c1/cf2, priority=9, startTime=1682175336793; duration=0sec 2023-04-22 14:55:36,835 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:36,835 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46-bottom 2023-04-22 14:55:36,836 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ca893cb83296e07cd3d89c3d9a1ef1c1/cf1 of ca893cb83296e07cd3d89c3d9a1ef1c1 into 1e79ad0f6ccb43a99b475895b8552fc3(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:36,836 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ca893cb83296e07cd3d89c3d9a1ef1c1: 2023-04-22 14:55:36,836 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1., storeName=ca893cb83296e07cd3d89c3d9a1ef1c1/cf1, priority=9, startTime=1682175336792; duration=0sec 2023-04-22 14:55:36,836 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:36,838 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41-bottom 2023-04-22 14:55:36,839 INFO [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(310): Store=b079ecd948cc7c7d6bd91b4d13027ab2/cf1, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:36,839 INFO [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf2 of region b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:36,839 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2 2023-04-22 14:55:36,839 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2 2023-04-22 14:55:36,840 INFO [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b079ecd948cc7c7d6bd91b4d13027ab2 columnFamilyName cf2 2023-04-22 14:55:36,851 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1-bottom 2023-04-22 14:55:36,854 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a-bottom 2023-04-22 14:55:36,857 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80-bottom 2023-04-22 14:55:36,859 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57-bottom 2023-04-22 14:55:36,862 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5-bottom 2023-04-22 14:55:36,864 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226-bottom 2023-04-22 14:55:36,867 DEBUG [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(539): loaded hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886-bottom 2023-04-22 14:55:36,867 INFO [StoreOpener-b079ecd948cc7c7d6bd91b4d13027ab2-1] regionserver.HStore(310): Store=b079ecd948cc7c7d6bd91b4d13027ab2/cf2, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-22 14:55:36,868 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:36,869 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:36,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table testSplitTableRegion descriptor;using region.getMemStoreFlushHeapSize/# of families (64.0 M)) instead. 2023-04-22 14:55:36,870 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:36,871 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b079ecd948cc7c7d6bd91b4d13027ab2; next sequenceid=86; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72659370, jitterRate=0.08270898461341858}}}, FlushLargeStoresPolicy{flushSizeLowerBound=67108864} 2023-04-22 14:55:36,871 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b079ecd948cc7c7d6bd91b4d13027ab2: 2023-04-22 14:55:36,872 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2., pid=149, masterSystemTime=1682175336724 2023-04-22 14:55:36,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:36,872 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: Opening Region; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-22 14:55:36,873 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:36,873 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-22 14:55:36,873 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf2 since it belongs to recently split daughter region testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:36,873 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1898): Keeping/Overriding Compaction request priority to -2147482648 for CF cf1 since it belongs to recently split daughter region testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:36,873 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1912): b079ecd948cc7c7d6bd91b4d13027ab2/cf1 is initiating minor compaction (all files) 2023-04-22 14:55:36,873 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1912): b079ecd948cc7c7d6bd91b4d13027ab2/cf2 is initiating minor compaction (all files) 2023-04-22 14:55:36,873 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b079ecd948cc7c7d6bd91b4d13027ab2/cf1 in testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:36,873 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b079ecd948cc7c7d6bd91b4d13027ab2/cf2 in testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:36,874 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/.tmp, totalSize=34.9 K 2023-04-22 14:55:36,874 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46-bottom] into tmpdir=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/.tmp, totalSize=34.9 K 2023-04-22 14:55:36,874 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:36,874 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175334921 2023-04-22 14:55:36,874 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:36,874 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027, keycount=1, bloomtype=ROW, size=4.8 K, encoding=NONE, compression=NONE, seqNum=6, earliestPutTs=1682175334921 2023-04-22 14:55:36,875 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=147 updating hbase:meta row=b079ecd948cc7c7d6bd91b4d13027ab2, regionState=OPEN, openSeqNum=86, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:36,875 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175335090 2023-04-22 14:55:36,875 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175336874"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1682175336874"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1682175336874"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1682175336874"}]},"ts":"1682175336874"} 2023-04-22 14:55:36,875 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=19, earliestPutTs=1682175335090 2023-04-22 14:55:36,876 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting 8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175335257 2023-04-22 14:55:36,876 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=32, earliestPutTs=1682175335257 2023-04-22 14:55:36,876 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175335425 2023-04-22 14:55:36,876 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1682175335425 2023-04-22 14:55:36,876 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175335599 2023-04-22 14:55:36,877 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175335765 2023-04-22 14:55:36,877 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1682175335599 2023-04-22 14:55:36,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=149, resume processing ppid=147 2023-04-22 14:55:36,877 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=147, state=SUCCESS; OpenRegionProcedure b079ecd948cc7c7d6bd91b4d13027ab2, server=jenkins-hbase4.apache.org,36267,1682175218583 in 303 msec 2023-04-22 14:55:36,877 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] compactions.Compactor(207): Compacting c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175335939 2023-04-22 14:55:36,878 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting 1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027, keycount=5, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=71, earliestPutTs=1682175335765 2023-04-22 14:55:36,878 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] compactions.Compactor(207): Compacting c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027, keycount=4, bloomtype=ROW, size=5.0 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1682175335939 2023-04-22 14:55:36,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=147, resume processing ppid=144 2023-04-22 14:55:36,879 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=147, ppid=144, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=b079ecd948cc7c7d6bd91b4d13027ab2, ASSIGN in 459 msec 2023-04-22 14:55:36,880 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=144, state=SUCCESS; SplitTableRegionProcedure table=testSplitTableRegion, parent=953631f39eb97e99d4a593add584f027, daughterA=b079ecd948cc7c7d6bd91b4d13027ab2, daughterB=ca893cb83296e07cd3d89c3d9a1ef1c1 in 933 msec 2023-04-22 14:55:36,886 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] throttle.PressureAwareThroughputController(145): b079ecd948cc7c7d6bd91b4d13027ab2#cf2#compaction#144 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:36,886 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] throttle.PressureAwareThroughputController(145): b079ecd948cc7c7d6bd91b4d13027ab2#cf1#compaction#145 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-22 14:55:36,907 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/.tmp/cf1/cb0213bfbba54a1aa914d18676089c63 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/cb0213bfbba54a1aa914d18676089c63 2023-04-22 14:55:36,908 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/.tmp/cf2/bcce468fec704e55ac3f0b20cc4b7774 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/bcce468fec704e55ac3f0b20cc4b7774 2023-04-22 14:55:36,912 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in b079ecd948cc7c7d6bd91b4d13027ab2/cf1 of b079ecd948cc7c7d6bd91b4d13027ab2 into cb0213bfbba54a1aa914d18676089c63(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:36,912 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b079ecd948cc7c7d6bd91b4d13027ab2: 2023-04-22 14:55:36,912 INFO [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2., storeName=b079ecd948cc7c7d6bd91b4d13027ab2/cf1, priority=9, startTime=1682175336872; duration=0sec 2023-04-22 14:55:36,912 DEBUG [RS:1;jenkins-hbase4:36267-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:36,913 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in b079ecd948cc7c7d6bd91b4d13027ab2/cf2 of b079ecd948cc7c7d6bd91b4d13027ab2 into bcce468fec704e55ac3f0b20cc4b7774(size=6.1 K), total size for store is 6.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-22 14:55:36,913 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b079ecd948cc7c7d6bd91b4d13027ab2: 2023-04-22 14:55:36,913 INFO [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2., storeName=b079ecd948cc7c7d6bd91b4d13027ab2/cf2, priority=9, startTime=1682175336872; duration=0sec 2023-04-22 14:55:36,913 DEBUG [RS:1;jenkins-hbase4:36267-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-22 14:55:36,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=36267] ipc.CallRunner(144): callId: 1120 service: ClientService methodName: Scan size: 143 connection: 172.31.14.131:50214 deadline: 1682175396947, exception=org.apache.hadoop.hbase.NotServingRegionException: testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027. is not online on jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:37,037 ERROR [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid 6cc12795-ed03-4ce7-985d-18d7c9a6ec2b) service to localhost/127.0.0.1:35843 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:37,059 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-22 14:55:37,066 INFO [Listener at localhost/43403] assignment.TestSplitTableRegionProcedure(570): Row Count = 60 2023-04-22 14:55:37,068 DEBUG [Listener at localhost/43403] client.ClientScanner(240): Advancing internal scanner to startKey at '41', inclusive 2023-04-22 14:55:37,074 WARN [Listener at localhost/43403] procedure2.ProcedureTestingUtility(193): Set Kill before store update to: false 2023-04-22 14:55:37,076 INFO [Listener at localhost/43403] client.HBaseAdmin$15(890): Started disable of testSplitTableRegion 2023-04-22 14:55:37,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable testSplitTableRegion 2023-04-22 14:55:37,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=151, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testSplitTableRegion 2023-04-22 14:55:37,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-04-22 14:55:37,079 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175337079"}]},"ts":"1682175337079"} 2023-04-22 14:55:37,080 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLING in hbase:meta 2023-04-22 14:55:37,082 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testSplitTableRegion to state=DISABLING 2023-04-22 14:55:37,083 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=b079ecd948cc7c7d6bd91b4d13027ab2, UNASSIGN}, {pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=ca893cb83296e07cd3d89c3d9a1ef1c1, UNASSIGN}] 2023-04-22 14:55:37,083 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=ca893cb83296e07cd3d89c3d9a1ef1c1, UNASSIGN 2023-04-22 14:55:37,084 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=ca893cb83296e07cd3d89c3d9a1ef1c1, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:37,084 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175337084"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175337084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175337084"}]},"ts":"1682175337084"} 2023-04-22 14:55:37,085 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=154, ppid=153, state=RUNNABLE; CloseRegionProcedure ca893cb83296e07cd3d89c3d9a1ef1c1, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:37,087 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=151, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testSplitTableRegion, region=b079ecd948cc7c7d6bd91b4d13027ab2, UNASSIGN 2023-04-22 14:55:37,087 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=b079ecd948cc7c7d6bd91b4d13027ab2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:37,087 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175337087"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1682175337087"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1682175337087"}]},"ts":"1682175337087"} 2023-04-22 14:55:37,090 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=152, state=RUNNABLE; CloseRegionProcedure b079ecd948cc7c7d6bd91b4d13027ab2, server=jenkins-hbase4.apache.org,36267,1682175218583}] 2023-04-22 14:55:37,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-04-22 14:55:37,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:37,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b079ecd948cc7c7d6bd91b4d13027ab2, disabling compactions & flushes 2023-04-22 14:55:37,237 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:37,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:37,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. after waiting 0 ms 2023-04-22 14:55:37,237 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:37,237 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46-bottom] to archive 2023-04-22 14:55:37,238 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:37,240 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,241 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,242 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,242 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,243 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,244 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,245 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,247 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80-bottom, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5-bottom] to archive 2023-04-22 14:55:37,247 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:37,248 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,249 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,250 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,251 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,252 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,253 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,253 DEBUG [StoreCloser-testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,256 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:55:37,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:37,257 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2. 2023-04-22 14:55:37,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b079ecd948cc7c7d6bd91b4d13027ab2: 2023-04-22 14:55:37,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:37,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:37,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ca893cb83296e07cd3d89c3d9a1ef1c1, disabling compactions & flushes 2023-04-22 14:55:37,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:37,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:37,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. after waiting 0 ms 2023-04-22 14:55:37,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:37,258 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=b079ecd948cc7c7d6bd91b4d13027ab2, regionState=CLOSED 2023-04-22 14:55:37,258 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175337258"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175337258"}]},"ts":"1682175337258"} 2023-04-22 14:55:37,259 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46-top] to archive 2023-04-22 14:55:37,260 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:37,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=152 2023-04-22 14:55:37,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=152, state=SUCCESS; CloseRegionProcedure b079ecd948cc7c7d6bd91b4d13027ab2, server=jenkins-hbase4.apache.org,36267,1682175218583 in 171 msec 2023-04-22 14:55:37,261 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=b079ecd948cc7c7d6bd91b4d13027ab2, UNASSIGN in 178 msec 2023-04-22 14:55:37,262 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/da9ab957c82e4db78955d94c0ec8ad41.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,263 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/3a7d8eb9d97d43cf824a628a587e5033.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,264 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/57e1413d85f749e0bbbf5ee90269d4c4.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,265 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/6852b591f3f8423bbde72c5d7711d24e.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,266 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/1c6f0a191e234fa29eb39538e704f5fb.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,267 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/c4074c83fec446079cb2f327b1b93f46.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,268 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80-top, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027->hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5-top] to archive 2023-04-22 14:55:37,269 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-22 14:55:37,270 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8909126e91094f838230b62af77056a1.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,271 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/f20c2072c13140ef97d30bd7a5bab886.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,272 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/8a1d7046d8cc46499032d76c974c6a4a.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,273 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/e691c55c7f0f4b7ca99f5f0e4271c226.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,273 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/bec8e24e22fc46fdbe734863ca69db57.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,274 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/b593c893effe4bb982f2d54e8b6a5f80.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,275 DEBUG [StoreCloser-testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5.953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/recovered.edits/90.seqid, newMaxSeqId=90, maxSeqId=85 2023-04-22 14:55:37,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:37,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1. 2023-04-22 14:55:37,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ca893cb83296e07cd3d89c3d9a1ef1c1: 2023-04-22 14:55:37,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:37,279 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=ca893cb83296e07cd3d89c3d9a1ef1c1, regionState=CLOSED 2023-04-22 14:55:37,279 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.","families":{"info":[{"qualifier":"regioninfo","vlen":56,"tag":[],"timestamp":"1682175337279"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1682175337279"}]},"ts":"1682175337279"} 2023-04-22 14:55:37,281 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=154, resume processing ppid=153 2023-04-22 14:55:37,281 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=153, state=SUCCESS; CloseRegionProcedure ca893cb83296e07cd3d89c3d9a1ef1c1, server=jenkins-hbase4.apache.org,36267,1682175218583 in 195 msec 2023-04-22 14:55:37,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=153, resume processing ppid=151 2023-04-22 14:55:37,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=151, state=SUCCESS; TransitRegionStateProcedure table=testSplitTableRegion, region=ca893cb83296e07cd3d89c3d9a1ef1c1, UNASSIGN in 198 msec 2023-04-22 14:55:37,283 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1682175337283"}]},"ts":"1682175337283"} 2023-04-22 14:55:37,284 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testSplitTableRegion, state=DISABLED in hbase:meta 2023-04-22 14:55:37,285 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testSplitTableRegion to state=DISABLED 2023-04-22 14:55:37,286 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=151, state=SUCCESS; DisableTableProcedure table=testSplitTableRegion in 210 msec 2023-04-22 14:55:37,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=151 2023-04-22 14:55:37,381 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testSplitTableRegion, procId: 151 completed 2023-04-22 14:55:37,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete testSplitTableRegion 2023-04-22 14:55:37,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] procedure2.ProcedureExecutor(1029): Stored pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testSplitTableRegion 2023-04-22 14:55:37,384 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=156, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-22 14:55:37,385 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=156, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-22 14:55:37,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=156 2023-04-22 14:55:37,388 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,388 DEBUG [HFileArchiver-15] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:37,388 DEBUG [HFileArchiver-14] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:37,389 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/recovered.edits] 2023-04-22 14:55:37,389 DEBUG [HFileArchiver-14] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/recovered.edits] 2023-04-22 14:55:37,390 DEBUG [HFileArchiver-15] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2, FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/recovered.edits] 2023-04-22 14:55:37,392 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/cb0213bfbba54a1aa914d18676089c63 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf1/cb0213bfbba54a1aa914d18676089c63 2023-04-22 14:55:37,392 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/1e79ad0f6ccb43a99b475895b8552fc3 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf1/1e79ad0f6ccb43a99b475895b8552fc3 2023-04-22 14:55:37,393 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/060e9aa7c7764a1ca13f04b47d4cc3e7 2023-04-22 14:55:37,394 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/1c6f0a191e234fa29eb39538e704f5fb 2023-04-22 14:55:37,394 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/b7a6dcf6e65940f3b380845f7beba9df to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/cf2/b7a6dcf6e65940f3b380845f7beba9df 2023-04-22 14:55:37,394 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/bcce468fec704e55ac3f0b20cc4b7774 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/cf2/bcce468fec704e55ac3f0b20cc4b7774 2023-04-22 14:55:37,395 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/3a7d8eb9d97d43cf824a628a587e5033 2023-04-22 14:55:37,397 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/57e1413d85f749e0bbbf5ee90269d4c4 2023-04-22 14:55:37,397 DEBUG [HFileArchiver-15] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1/recovered.edits/90.seqid 2023-04-22 14:55:37,397 DEBUG [HFileArchiver-14] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/recovered.edits/90.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2/recovered.edits/90.seqid 2023-04-22 14:55:37,397 DEBUG [HFileArchiver-15] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/ca893cb83296e07cd3d89c3d9a1ef1c1 2023-04-22 14:55:37,397 DEBUG [HFileArchiver-14] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/b079ecd948cc7c7d6bd91b4d13027ab2 2023-04-22 14:55:37,398 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/6852b591f3f8423bbde72c5d7711d24e 2023-04-22 14:55:37,399 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/c4074c83fec446079cb2f327b1b93f46 2023-04-22 14:55:37,400 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf1/da9ab957c82e4db78955d94c0ec8ad41 2023-04-22 14:55:37,402 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8909126e91094f838230b62af77056a1 2023-04-22 14:55:37,402 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/8a1d7046d8cc46499032d76c974c6a4a 2023-04-22 14:55:37,403 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/b593c893effe4bb982f2d54e8b6a5f80 2023-04-22 14:55:37,404 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/bec8e24e22fc46fdbe734863ca69db57 2023-04-22 14:55:37,405 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/c84edff8e53c4fbf8a0a1dbf2ccf2cd5 2023-04-22 14:55:37,406 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/e691c55c7f0f4b7ca99f5f0e4271c226 2023-04-22 14:55:37,407 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886 to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/cf2/f20c2072c13140ef97d30bd7a5bab886 2023-04-22 14:55:37,409 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/recovered.edits/85.seqid to hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/archive/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027/recovered.edits/85.seqid 2023-04-22 14:55:37,409 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/.tmp/data/default/testSplitTableRegion/953631f39eb97e99d4a593add584f027 2023-04-22 14:55:37,409 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testSplitTableRegion regions 2023-04-22 14:55:37,411 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=156, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-22 14:55:37,412 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 3 rows of testSplitTableRegion from hbase:meta 2023-04-22 14:55:37,414 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testSplitTableRegion' descriptor. 2023-04-22 14:55:37,414 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=156, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-22 14:55:37,414 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testSplitTableRegion' from region states. 2023-04-22 14:55:37,415 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175337414"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:37,415 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175337414"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:37,415 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1682175337414"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:37,416 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 3 regions from META 2023-04-22 14:55:37,416 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 953631f39eb97e99d4a593add584f027, NAME => 'testSplitTableRegion,,1682175334418.953631f39eb97e99d4a593add584f027.', STARTKEY => '', ENDKEY => ''}, {ENCODED => b079ecd948cc7c7d6bd91b4d13027ab2, NAME => 'testSplitTableRegion,,1682175335946.b079ecd948cc7c7d6bd91b4d13027ab2.', STARTKEY => '', ENDKEY => '41'}, {ENCODED => ca893cb83296e07cd3d89c3d9a1ef1c1, NAME => 'testSplitTableRegion,41,1682175335946.ca893cb83296e07cd3d89c3d9a1ef1c1.', STARTKEY => '41', ENDKEY => ''}] 2023-04-22 14:55:37,416 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testSplitTableRegion' as deleted. 2023-04-22 14:55:37,416 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testSplitTableRegion","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1682175337416"}]},"ts":"9223372036854775807"} 2023-04-22 14:55:37,417 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testSplitTableRegion state from META 2023-04-22 14:55:37,418 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=156, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testSplitTableRegion 2023-04-22 14:55:37,419 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=156, state=SUCCESS; DeleteTableProcedure table=testSplitTableRegion in 37 msec 2023-04-22 14:55:37,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43497] master.MasterRpcServices(1227): Checking to see if procedure is done pid=156 2023-04-22 14:55:37,486 INFO [Listener at localhost/43403] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testSplitTableRegion, procId: 156 completed 2023-04-22 14:55:37,500 INFO [Listener at localhost/43403] hbase.ResourceChecker(175): after: master.assignment.TestSplitTableRegionProcedure#testSplitTableRegion Thread=377 (was 372) - Thread LEAK? -, OpenFileDescriptor=850 (was 835) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=314 (was 342), ProcessCount=169 (was 170), AvailableMemoryMB=12321 (was 12368) 2023-04-22 14:55:37,500 INFO [Listener at localhost/43403] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-22 14:55:37,500 INFO [Listener at localhost/43403] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-22 14:55:37,501 DEBUG [Listener at localhost/43403] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e0ba461 to 127.0.0.1:50713 2023-04-22 14:55:37,501 DEBUG [Listener at localhost/43403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:37,501 DEBUG [Listener at localhost/43403] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-22 14:55:37,502 DEBUG [Listener at localhost/43403] util.JVMClusterUtil(257): Found active master hash=576052752, stopped=false 2023-04-22 14:55:37,502 INFO [Listener at localhost/43403] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:55:37,504 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-22 14:55:37,504 INFO [Listener at localhost/43403] procedure2.ProcedureExecutor(629): Stopping 2023-04-22 14:55:37,504 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-22 14:55:37,504 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:55:37,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:55:37,504 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-22 14:55:37,504 DEBUG [Listener at localhost/43403] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27b5d011 to 127.0.0.1:50713 2023-04-22 14:55:37,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:55:37,504 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-22 14:55:37,505 DEBUG [Listener at localhost/43403] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:37,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:55:37,505 INFO [Listener at localhost/43403] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35417,1682175218529' ***** 2023-04-22 14:55:37,505 INFO [Listener at localhost/43403] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-22 14:55:37,505 INFO [Listener at localhost/43403] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,36267,1682175218583' ***** 2023-04-22 14:55:37,505 INFO [Listener at localhost/43403] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-22 14:55:37,505 INFO [Listener at localhost/43403] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33597,1682175218631' ***** 2023-04-22 14:55:37,505 INFO [Listener at localhost/43403] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-22 14:55:37,505 INFO [RS:2;jenkins-hbase4:33597] regionserver.HeapMemoryManager(220): Stopping 2023-04-22 14:55:37,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-22 14:55:37,505 INFO [RS:0;jenkins-hbase4:35417] regionserver.HeapMemoryManager(220): Stopping 2023-04-22 14:55:37,506 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-22 14:55:37,506 INFO [RS:0;jenkins-hbase4:35417] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-22 14:55:37,506 INFO [RS:2;jenkins-hbase4:33597] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-22 14:55:37,506 INFO [RS:0;jenkins-hbase4:35417] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-22 14:55:37,506 INFO [RS:2;jenkins-hbase4:33597] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-22 14:55:37,506 INFO [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:37,506 INFO [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:37,506 INFO [RS:1;jenkins-hbase4:36267] regionserver.HeapMemoryManager(220): Stopping 2023-04-22 14:55:37,506 DEBUG [RS:2;jenkins-hbase4:33597] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x359d0571 to 127.0.0.1:50713 2023-04-22 14:55:37,506 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-22 14:55:37,506 DEBUG [RS:0;jenkins-hbase4:35417] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x386efff5 to 127.0.0.1:50713 2023-04-22 14:55:37,506 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-22 14:55:37,506 DEBUG [RS:0;jenkins-hbase4:35417] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:37,506 INFO [RS:1;jenkins-hbase4:36267] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-22 14:55:37,506 INFO [RS:0;jenkins-hbase4:35417] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-22 14:55:37,507 INFO [RS:0;jenkins-hbase4:35417] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-22 14:55:37,507 INFO [RS:0;jenkins-hbase4:35417] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-22 14:55:37,506 DEBUG [RS:2;jenkins-hbase4:33597] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:37,507 INFO [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-22 14:55:37,507 INFO [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33597,1682175218631; all regions closed. 2023-04-22 14:55:37,506 INFO [RS:1;jenkins-hbase4:36267] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-22 14:55:37,507 INFO [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(3304): Received CLOSE for d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:55:37,507 INFO [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:37,507 DEBUG [RS:1;jenkins-hbase4:36267] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e7fce47 to 127.0.0.1:50713 2023-04-22 14:55:37,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d65d90b6fbc4a32d4609259a87039a65, disabling compactions & flushes 2023-04-22 14:55:37,507 INFO [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-22 14:55:37,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:55:37,507 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-22 14:55:37,507 DEBUG [RS:1;jenkins-hbase4:36267] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:37,507 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-22 14:55:37,507 INFO [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-22 14:55:37,507 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-22 14:55:37,507 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1479): Online Regions={d65d90b6fbc4a32d4609259a87039a65=hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65.} 2023-04-22 14:55:37,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:55:37,507 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-22 14:55:37,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. after waiting 0 ms 2023-04-22 14:55:37,507 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-22 14:55:37,508 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-22 14:55:37,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:55:37,663 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d65d90b6fbc4a32d4609259a87039a65 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-22 14:55:37,662 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-22 14:55:37,662 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-22 14:55:37,663 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=79.40 KB heapSize=125.77 KB 2023-04-22 14:55:37,662 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-22 14:55:37,663 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1505): Waiting on d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:55:37,663 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 14:55:37,677 DEBUG [RS:2;jenkins-hbase4:33597] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/oldWALs 2023-04-22 14:55:37,677 INFO [RS:2;jenkins-hbase4:33597] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C33597%2C1682175218631:(num 1682175220276) 2023-04-22 14:55:37,677 DEBUG [RS:2;jenkins-hbase4:33597] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:37,677 INFO [RS:2;jenkins-hbase4:33597] regionserver.LeaseManager(133): Closed leases 2023-04-22 14:55:37,677 INFO [RS:2;jenkins-hbase4:33597] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-22 14:55:37,678 INFO [RS:2;jenkins-hbase4:33597] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-22 14:55:37,678 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-22 14:55:37,678 INFO [RS:2;jenkins-hbase4:33597] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-22 14:55:37,678 INFO [RS:2;jenkins-hbase4:33597] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-22 14:55:37,680 INFO [RS:2;jenkins-hbase4:33597] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33597 2023-04-22 14:55:37,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:37,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:55:37,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:37,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,33597,1682175218631 2023-04-22 14:55:37,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:55:37,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:55:37,686 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:55:37,686 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@706b4ee9 rejected from java.util.concurrent.ThreadPoolExecutor@7454d43c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:37,687 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,33597,1682175218631] 2023-04-22 14:55:37,687 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,33597,1682175218631; numProcessing=1 2023-04-22 14:55:37,688 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,33597,1682175218631 already deleted, retry=false 2023-04-22 14:55:37,688 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,33597,1682175218631 expired; onlineServers=2 2023-04-22 14:55:37,787 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:55:37,787 INFO [RS:2;jenkins-hbase4:33597] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33597,1682175218631; zookeeper connection closed. 2023-04-22 14:55:37,787 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2a1bdb6b rejected from java.util.concurrent.ThreadPoolExecutor@7454d43c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:37,787 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:33597-0x101ac5cb4e00003, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:55:37,788 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@520a15da rejected from java.util.concurrent.ThreadPoolExecutor@7454d43c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 310] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:37,788 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@235e7bfd] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@235e7bfd 2023-04-22 14:55:37,863 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 14:55:37,863 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1505): Waiting on d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:55:38,035 ERROR [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(848): Exception in BPOfferService for Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid cd7c3aae-8b07-4809-9172-b4e8dd2fdabd) service to localhost/127.0.0.1:35843 java.lang.IllegalThreadStateException at java.lang.ThreadGroup.addUnstarted(ThreadGroup.java:867) at java.lang.Thread.init(Thread.java:407) at java.lang.Thread.init(Thread.java:351) at java.lang.Thread.(Thread.java:501) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService$1.newThread(FsDatasetAsyncDiskService.java:102) at java.util.concurrent.ThreadPoolExecutor$Worker.(ThreadPoolExecutor.java:619) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:932) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1367) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.execute(FsDatasetAsyncDiskService.java:172) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService.deleteAsync(FsDatasetAsyncDiskService.java:221) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2216) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.invalidate(FsDatasetImpl.java:2140) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:732) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:679) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processCommand(BPServiceActor.java:880) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:676) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846) at java.lang.Thread.run(Thread.java:750) 2023-04-22 14:55:38,064 DEBUG [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-22 14:55:38,064 DEBUG [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1505): Waiting on d65d90b6fbc4a32d4609259a87039a65 2023-04-22 14:55:38,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65/.tmp/info/edd804d48ef54480b558b371107f31e9 2023-04-22 14:55:38,083 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.29 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/.tmp/info/fd9cc746f3aa435e9af7987697140de9 2023-04-22 14:55:38,087 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fd9cc746f3aa435e9af7987697140de9 2023-04-22 14:55:38,088 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65/.tmp/info/edd804d48ef54480b558b371107f31e9 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65/info/edd804d48ef54480b558b371107f31e9 2023-04-22 14:55:38,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65/info/edd804d48ef54480b558b371107f31e9, entries=2, sequenceid=6, filesize=4.8 K 2023-04-22 14:55:38,092 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for d65d90b6fbc4a32d4609259a87039a65 in 429ms, sequenceid=6, compaction requested=false 2023-04-22 14:55:38,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/namespace/d65d90b6fbc4a32d4609259a87039a65/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-22 14:55:38,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:38,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:55:38,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d65d90b6fbc4a32d4609259a87039a65: 2023-04-22 14:55:38,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1682175220771.d65d90b6fbc4a32d4609259a87039a65. 2023-04-22 14:55:38,099 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.58 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/.tmp/rep_barrier/d6a1ca7d34934858aa6d1b9e8f1afb7b 2023-04-22 14:55:38,102 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d6a1ca7d34934858aa6d1b9e8f1afb7b 2023-04-22 14:55:38,108 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-22 14:55:38,108 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-22 14:55:38,111 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=5.53 KB at sequenceid=210 (bloomFilter=false), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/.tmp/table/82b1122d91944bb08a33f42d046b4589 2023-04-22 14:55:38,115 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 82b1122d91944bb08a33f42d046b4589 2023-04-22 14:55:38,116 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/.tmp/info/fd9cc746f3aa435e9af7987697140de9 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/info/fd9cc746f3aa435e9af7987697140de9 2023-04-22 14:55:38,119 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fd9cc746f3aa435e9af7987697140de9 2023-04-22 14:55:38,119 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/info/fd9cc746f3aa435e9af7987697140de9, entries=59, sequenceid=210, filesize=11.1 K 2023-04-22 14:55:38,120 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/.tmp/rep_barrier/d6a1ca7d34934858aa6d1b9e8f1afb7b as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/rep_barrier/d6a1ca7d34934858aa6d1b9e8f1afb7b 2023-04-22 14:55:38,124 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d6a1ca7d34934858aa6d1b9e8f1afb7b 2023-04-22 14:55:38,124 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/rep_barrier/d6a1ca7d34934858aa6d1b9e8f1afb7b, entries=24, sequenceid=210, filesize=7.5 K 2023-04-22 14:55:38,124 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/.tmp/table/82b1122d91944bb08a33f42d046b4589 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/table/82b1122d91944bb08a33f42d046b4589 2023-04-22 14:55:38,128 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 82b1122d91944bb08a33f42d046b4589 2023-04-22 14:55:38,128 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/table/82b1122d91944bb08a33f42d046b4589, entries=36, sequenceid=210, filesize=8.0 K 2023-04-22 14:55:38,129 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~79.40 KB/81308, heapSize ~125.73 KB/128744, currentSize=0 B/0 for 1588230740 in 621ms, sequenceid=210, compaction requested=false 2023-04-22 14:55:38,133 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/data/hbase/meta/1588230740/recovered.edits/213.seqid, newMaxSeqId=213, maxSeqId=1 2023-04-22 14:55:38,134 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.master.assignment.TestSplitTableRegionProcedure$RegionServerHostingReplicaSlowOpenCopro 2023-04-22 14:55:38,134 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-22 14:55:38,134 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-22 14:55:38,134 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-22 14:55:38,135 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-22 14:55:38,188 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-22 14:55:38,188 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-22 14:55:38,264 INFO [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35417,1682175218529; all regions closed. 2023-04-22 14:55:38,264 INFO [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,36267,1682175218583; all regions closed. 2023-04-22 14:55:38,269 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/WALs/jenkins-hbase4.apache.org,36267,1682175218583/jenkins-hbase4.apache.org%2C36267%2C1682175218583.1682175220282 not finished, retry = 0 2023-04-22 14:55:38,271 DEBUG [RS:0;jenkins-hbase4:35417] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/oldWALs 2023-04-22 14:55:38,271 INFO [RS:0;jenkins-hbase4:35417] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35417%2C1682175218529.meta:.meta(num 1682175220463) 2023-04-22 14:55:38,276 DEBUG [RS:0;jenkins-hbase4:35417] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/oldWALs 2023-04-22 14:55:38,276 INFO [RS:0;jenkins-hbase4:35417] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35417%2C1682175218529:(num 1682175220276) 2023-04-22 14:55:38,276 DEBUG [RS:0;jenkins-hbase4:35417] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:38,276 INFO [RS:0;jenkins-hbase4:35417] regionserver.LeaseManager(133): Closed leases 2023-04-22 14:55:38,276 INFO [RS:0;jenkins-hbase4:35417] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-04-22 14:55:38,277 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-22 14:55:38,278 INFO [RS:0;jenkins-hbase4:35417] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35417 2023-04-22 14:55:38,281 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:38,281 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35417,1682175218529 2023-04-22 14:55:38,281 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5c5b37c6 rejected from java.util.concurrent.ThreadPoolExecutor@5864b895[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:38,281 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:55:38,282 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35417,1682175218529] 2023-04-22 14:55:38,282 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35417,1682175218529; numProcessing=2 2023-04-22 14:55:38,283 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35417,1682175218529 already deleted, retry=false 2023-04-22 14:55:38,283 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35417,1682175218529 expired; onlineServers=1 2023-04-22 14:55:38,372 DEBUG [RS:1;jenkins-hbase4:36267] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/oldWALs 2023-04-22 14:55:38,372 INFO [RS:1;jenkins-hbase4:36267] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C36267%2C1682175218583:(num 1682175220282) 2023-04-22 14:55:38,372 DEBUG [RS:1;jenkins-hbase4:36267] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:38,372 INFO [RS:1;jenkins-hbase4:36267] regionserver.LeaseManager(133): Closed leases 2023-04-22 14:55:38,372 INFO [RS:1;jenkins-hbase4:36267] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-22 14:55:38,372 INFO [RS:1;jenkins-hbase4:36267] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-22 14:55:38,372 INFO [RS:1;jenkins-hbase4:36267] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-22 14:55:38,372 INFO [RS:1;jenkins-hbase4:36267] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-22 14:55:38,372 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-22 14:55:38,373 INFO [RS:1;jenkins-hbase4:36267] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:36267 2023-04-22 14:55:38,375 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-22 14:55:38,375 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,36267,1682175218583 2023-04-22 14:55:38,375 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@f213e80 rejected from java.util.concurrent.ThreadPoolExecutor@b1eabbc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:38,377 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,36267,1682175218583] 2023-04-22 14:55:38,377 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,36267,1682175218583; numProcessing=3 2023-04-22 14:55:38,378 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,36267,1682175218583 already deleted, retry=false 2023-04-22 14:55:38,378 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,36267,1682175218583 expired; onlineServers=0 2023-04-22 14:55:38,378 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43497,1682175217266' ***** 2023-04-22 14:55:38,378 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-22 14:55:38,379 DEBUG [M:0;jenkins-hbase4:43497] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6368d42b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-22 14:55:38,379 INFO [M:0;jenkins-hbase4:43497] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43497,1682175217266 2023-04-22 14:55:38,379 INFO [M:0;jenkins-hbase4:43497] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43497,1682175217266; all regions closed. 2023-04-22 14:55:38,379 DEBUG [M:0;jenkins-hbase4:43497] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-22 14:55:38,379 DEBUG [M:0;jenkins-hbase4:43497] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-22 14:55:38,379 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-22 14:55:38,379 DEBUG [M:0;jenkins-hbase4:43497] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-22 14:55:38,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682175219917] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1682175219917,5,FailOnTimeoutGroup] 2023-04-22 14:55:38,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682175219917] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1682175219917,5,FailOnTimeoutGroup] 2023-04-22 14:55:38,379 INFO [M:0;jenkins-hbase4:43497] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-22 14:55:38,379 INFO [M:0;jenkins-hbase4:43497] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-22 14:55:38,379 INFO [M:0;jenkins-hbase4:43497] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-22 14:55:38,379 DEBUG [M:0;jenkins-hbase4:43497] master.HMaster(1512): Stopping service threads 2023-04-22 14:55:38,379 INFO [M:0;jenkins-hbase4:43497] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-22 14:55:38,380 ERROR [M:0;jenkins-hbase4:43497] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] Thread[HFileArchiver-14,5,PEWorkerGroup] Thread[HFileArchiver-15,5,PEWorkerGroup] Thread[HFileArchiver-16,5,PEWorkerGroup] 2023-04-22 14:55:38,380 INFO [M:0;jenkins-hbase4:43497] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-22 14:55:38,380 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-22 14:55:38,380 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-22 14:55:38,380 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-22 14:55:38,380 DEBUG [M:0;jenkins-hbase4:43497] zookeeper.ZKUtil(398): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-22 14:55:38,380 WARN [M:0;jenkins-hbase4:43497] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-22 14:55:38,380 INFO [M:0;jenkins-hbase4:43497] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-22 14:55:38,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-22 14:55:38,380 INFO [M:0;jenkins-hbase4:43497] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-22 14:55:38,381 DEBUG [M:0;jenkins-hbase4:43497] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-22 14:55:38,381 INFO [M:0;jenkins-hbase4:43497] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:55:38,381 DEBUG [M:0;jenkins-hbase4:43497] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:55:38,381 DEBUG [M:0;jenkins-hbase4:43497] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-22 14:55:38,381 DEBUG [M:0;jenkins-hbase4:43497] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:55:38,381 INFO [M:0;jenkins-hbase4:43497] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=607.94 KB heapSize=718.95 KB 2023-04-22 14:55:38,382 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:55:38,382 INFO [RS:0;jenkins-hbase4:35417] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35417,1682175218529; zookeeper connection closed. 2023-04-22 14:55:38,382 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2951c64f rejected from java.util.concurrent.ThreadPoolExecutor@5864b895[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:38,382 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:35417-0x101ac5cb4e00001, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:55:38,382 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6d6406bf] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6d6406bf 2023-04-22 14:55:38,382 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@51ec84a7 rejected from java.util.concurrent.ThreadPoolExecutor@5864b895[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 312] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:38,391 INFO [M:0;jenkins-hbase4:43497] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=607.94 KB at sequenceid=1282 (bloomFilter=true), to=hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d495503071eb44ac961340932e8bca41 2023-04-22 14:55:38,394 INFO [M:0;jenkins-hbase4:43497] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d495503071eb44ac961340932e8bca41 2023-04-22 14:55:38,395 DEBUG [M:0;jenkins-hbase4:43497] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/d495503071eb44ac961340932e8bca41 as hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d495503071eb44ac961340932e8bca41 2023-04-22 14:55:38,398 INFO [M:0;jenkins-hbase4:43497] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for d495503071eb44ac961340932e8bca41 2023-04-22 14:55:38,398 INFO [M:0;jenkins-hbase4:43497] regionserver.HStore(1080): Added hdfs://localhost:35843/user/jenkins/test-data/9eec8fbf-8eba-d259-986f-75c7030c52fd/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/d495503071eb44ac961340932e8bca41, entries=156, sequenceid=1282, filesize=34.9 K 2023-04-22 14:55:38,398 INFO [M:0;jenkins-hbase4:43497] regionserver.HRegion(2948): Finished flush of dataSize ~607.94 KB/622533, heapSize ~718.93 KB/736184, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 17ms, sequenceid=1282, compaction requested=false 2023-04-22 14:55:38,399 INFO [M:0;jenkins-hbase4:43497] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-22 14:55:38,399 DEBUG [M:0;jenkins-hbase4:43497] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-22 14:55:38,403 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-22 14:55:38,403 INFO [M:0;jenkins-hbase4:43497] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-22 14:55:38,404 INFO [M:0;jenkins-hbase4:43497] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43497 2023-04-22 14:55:38,406 DEBUG [M:0;jenkins-hbase4:43497] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,43497,1682175217266 already deleted, retry=false 2023-04-22 14:55:38,482 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:55:38,482 INFO [RS:1;jenkins-hbase4:36267] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,36267,1682175218583; zookeeper connection closed. 2023-04-22 14:55:38,482 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@79cb6d14 rejected from java.util.concurrent.ThreadPoolExecutor@b1eabbc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:38,483 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): regionserver:36267-0x101ac5cb4e00002, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:55:38,483 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1a59e97 rejected from java.util.concurrent.ThreadPoolExecutor@b1eabbc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 313] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:38,483 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@498630ab] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@498630ab 2023-04-22 14:55:38,483 INFO [Listener at localhost/43403] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-04-22 14:55:38,583 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:55:38,583 INFO [M:0;jenkins-hbase4:43497] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43497,1682175217266; zookeeper connection closed. 2023-04-22 14:55:38,583 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3ef4d39a rejected from java.util.concurrent.ThreadPoolExecutor@68799a52[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 752] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:38,583 DEBUG [Listener at localhost/43403-EventThread] zookeeper.ZKWatcher(600): master:43497-0x101ac5cb4e00000, quorum=127.0.0.1:50713, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-22 14:55:38,583 ERROR [Listener at localhost/43403-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4f02991a rejected from java.util.concurrent.ThreadPoolExecutor@68799a52[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 752] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-22 14:55:38,585 WARN [Listener at localhost/43403] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-22 14:55:38,588 INFO [Listener at localhost/43403] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-22 14:55:38,691 WARN [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-22 14:55:38,691 WARN [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid 0e9964a3-eb0e-4f0a-bd76-2c6599875fd5) service to localhost/127.0.0.1:35843 2023-04-22 14:55:38,694 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data5/current/BP-1485136854-172.31.14.131-1682175213588] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 14:55:38,694 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data6/current/BP-1485136854-172.31.14.131-1682175213588] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 14:55:38,696 WARN [Listener at localhost/43403] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-22 14:55:38,698 INFO [Listener at localhost/43403] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-22 14:55:38,804 WARN [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid 6cc12795-ed03-4ce7-985d-18d7c9a6ec2b) service to localhost/127.0.0.1:35843 2023-04-22 14:55:38,805 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data3/current/BP-1485136854-172.31.14.131-1682175213588] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 14:55:38,805 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data4/current/BP-1485136854-172.31.14.131-1682175213588] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 14:55:38,807 WARN [Listener at localhost/43403] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-22 14:55:38,811 INFO [Listener at localhost/43403] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-22 14:55:38,915 WARN [BP-1485136854-172.31.14.131-1682175213588 heartbeating to localhost/127.0.0.1:35843] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1485136854-172.31.14.131-1682175213588 (Datanode Uuid cd7c3aae-8b07-4809-9172-b4e8dd2fdabd) service to localhost/127.0.0.1:35843 2023-04-22 14:55:38,917 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data1/current/BP-1485136854-172.31.14.131-1682175213588] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 14:55:38,918 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/debd42d0-d920-ffff-21a5-debc9e9288a3/cluster_d3e6696b-2d5f-ec0c-d050-011332bf5a1a/dfs/data/data2/current/BP-1485136854-172.31.14.131-1682175213588] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-22 14:55:38,939 INFO [Listener at localhost/43403] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-22 14:55:39,131 INFO [Listener at localhost/43403] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-22 14:55:39,364 INFO [Listener at localhost/43403] hbase.HBaseTestingUtility(1293): Minicluster is down